Alexandria: A Multi-Domain Dialectal Arabic Machine Translation Dataset for Culturally Inclusive and Linguistically Diverse LLMs

AI summary

Overview

Research area: Natural Language Processing, specifically machine translation and multilingual large language model evaluation for Arabic.

Technical level: Intermediate. The paper is accessible to readers familiar with MT metrics and LLM evaluation concepts, but the core contribution (dataset construction) is largely descriptive rather than mathematically dense.

Scope: The paper introduces and benchmarks Alexandria, a large-scale human-translated parallel corpus of English and dialectal Arabic multi-turn conversations spanning 13 Arab countries, 11 domains, and city-level sub-dialects.

What This Paper Is About

Arabic is diglossic: formal writing uses Modern Standard Arabic (MSA), but everyday speech happens in regional dialects that differ substantially across countries and even between cities. Machine translation systems trained mostly on MSA or English data generalize poorly to these dialects, so millions of speakers are poorly served. The paper's goal is to build a large, culturally grounded, human-translated dataset that both trains and rigorously evaluates MT and LLM systems on dialectal Arabic.

Key Contributions

1. The Alexandria dataset: 34,488 multi-turn conversational scenarios totaling 107K turns, covering 13 Arab countries and 11 high-impact domains (health, education, agriculture, legal, tourism, and others), with parallel English–Dialectal Arabic translations.

2. Unprecedented granularity: Instead of coarse regional labels ("Levantine," "Gulf"), each contribution is tagged with city-of-origin metadata, capturing authentic local sub-dialects. Each turn also carries speaker-addressee gender configuration (F→M, M→F, M→M, F→F) and persona/role information for studying gender-conditioned and register-conditioned variation.

3. A community-driven creation protocol: 55 self-identified native speakers (29 women, 26 men) across the 13 countries produced the data over six months, with published generation prompts, translation guidelines, and revision rubrics.

4. A rigorous evaluation suite: Automatic (spBLEU, chrF++) and human evaluation of 24 Arabic-capable LLMs plus an NLLB-200-3.3B baseline across three input settings (turn-level, context-level, conversation-level), including ablations on metadata conditioning, code-switching, MSA distance, and reasoning/thinking modes.

Main Findings

• Strong directional asymmetry: Dialect→English translation is consistently easier than English→Dialect across all models; the latter is more sensitive to dialect authenticity and MSA leakage.

• Dialect difficulty is uneven: Egyptian and Levantine varieties (Syrian, Lebanese, Jordanian) score highest, likely reflecting greater training-data coverage. Maghrebi varieties are hardest, with Mauritanian yielding the lowest scores.

• Model leaders: Gemini-2.5-Pro and Gemini-3-Flash achieve the strongest performance in both directions, followed by Command A. Smaller open-weight models such as ALLaM-7B and Fanar-1.9B occupy the lower tier.

• MSA proximity predicts performance: Lexical overlap between dialectal references and NLLB-generated MSA correlates positively with spBLEU (up to 0.48 for Saudi, 0.44 for Yemeni), confirming that dialects structurally closer to MSA are easier to translate.

• Code-switching hurts: Sentences containing Latin-script borrowings (English, French, Spanish) yield noticeably lower spBLEU for most dialects — Egyptian, Jordanian, Lebanese, Moroccan, Palestinian, and Tunisian — with negative correlations against the Code-Mixing Index.

• Metadata effects are mixed: Full persona/gender metadata helps some models (e.g., c4ai-command-a) but can hurt others (e.g., aya-expanse-32b drops sharply with full metadata). Gains are model- and dialect-dependent, not monotonic.

• Reasoning rarely helps: Enabling "thinking" modes degrades translation in most cases; only Gemini-3-Flash benefits, gaining roughly +2.0 spBLEU for English→Dialect.

• Human evaluation exposes a gap: Gender accuracy is high (typically >90%), but semantic adequacy averages above 3/5 while dialectness/fluency drops to roughly 2/5 for some model-country pairs. Most points fall below the identity line, meaning models preserve meaning far better than they produce authentic dialect.

• Domain robustness: Model rankings are highly stable across the 11 domains, suggesting overall model strength predicts performance better than domain specialization.

• Dataset quality benchmarks: During revision, 68.4% of turns were accepted unchanged, 30.6% received minor edits, and 1% were flagged for major issues. Final quality ratings averaged 9.03/10 for dialectal authenticity, 9.40/10 for register, and 9.36/10 for semantic faithfulness.

• Gender imbalance in data: Final gender distribution is F→M 33.19%, M→F 32.78%, M→M 21.43%, F→F 12.60%, with the female-to-female scenario underrepresented due to the generation pipeline.

Methodology in Plain English

The team ran a six-month, community-driven pipeline in three phases.

Phase one — English sources. Using Gemini-2.5 Pro with country- and domain-specific prompts, they first generated 55 subdomains per country-domain pair and 10 topics per subdomain (550 topics total), each tied to a persona profile with role and gender attributes. They then generated 2–4 turn English dialogues grounded in the target country's culture, excluding personally identifiable information and deliberately using English paraphrases for culturally specific terms (e.g., "God willing" rather than "inshallah") to avoid transliteration bias. This produced roughly 6,050 conversations per country.

Phase two — Human translation. Native speakers from each country translated each turn into their local dialect using Arabic script, preserving semantics, persona, speaker-addressee gender, and appropriate register. They were allowed to use conventional borrowed terms in Latin script where no natural dialectal equivalent exists. AI assistance was permitted only with strict human post-editing.

Phase three — Peer revision. A second participant from the same country reviewed each conversation along six dimensions: dialect authenticity, gender alignment, register, semantic faithfulness, punctuation, and code-switching consistency. Reviewers could accept, make minor mechanical edits, or flag major issues for escalation to the country lead. Reviewers were explicitly barred from altering dialect-specific phrasing when the translation used a variety different from their own.

Evaluation. The public test set was used to benchmark 24 Arabic-capable LLMs under three settings: translating a single turn in isolation, translating a turn with preceding dialogue context, and translating the whole conversation at once. Metrics were spBLEU and chrF++, with human evaluation on semantic adequacy (5-point XSTS scale), gender accuracy (Pass/Fail/NA), and dialectness/fluency (1–5), rated by native speakers across countries.

Why This Matters

This work directly targets a structural equity gap in language technology: Arabic speakers who communicate daily in dialects are systematically underserved by MT and LLM systems optimized for MSA and high-resource languages. Alexandria's scale, domain diversity, and city-level granularity make it the most fine-grained resource of its kind, and its explicit gender and register annotations open new research questions about culturally conditioned language variation.

Real-world applications include:

• Healthcare access: Reliable dialectal translation of medical conversations for patients who speak only local Arabic varieties.
• Government and legal services: Multilingual public service delivery for refugees, migrants, and citizens across Arab countries.
• Education and agriculture extension: Adapting technical content into locally intelligible dialect for rural and low-literacy populations.
• Customer service and tourism: Chatbot and translation systems that produce culturally plausible, dialect-authentic responses.

Industry relevance: Any company deploying LLMs, MT engines, voice assistants, or customer support in the Arab world (telecoms, banking, e-commerce, humanitarian NGOs, government digitization) will find this benchmark directly relevant. It also provides a gold standard for evaluating whether a model's Arabic output is actually native-sounding or just MSA in disguise — a failure mode that silently erodes user trust.

Future Directions

• Fix the gender imbalance: Increase female-to-female conversational scenarios, which were unintentionally suppressed by the source-generation pipeline, and study how this affects gender-conditioned translation quality.

• Close the dialectness gap: Models preserve meaning but rarely produce authentic dialect. Future work should focus on decoding, prompting, or fine-tuning strategies that raise dialectness without sacrificing adequacy.

• Understand context conditioning: Metadata helps inconsistently. More systematic study is needed on how personas, register cues, and speaker-addressee pairs interact with model architecture and dialect.

• Extend coverage and modalities: Expand to additional sub-dialects (more cities within each country), handle code-switching more gracefully, and explore speech-based dialectal translation for spoken conversation.

Target Audience

This paper benefits NLP and MT researchers working on low-resource and dialectal languages, Arabic dialectologists studying regional variation and gender-conditioned speech, LLM practitioners building Arabic-facing applications, and localization or humanitarian technology teams that need realistic dialectal data. It is also useful for evaluation researchers interested in benchmark design that goes beyond coarse regional labels and surface metrics.

Authors’ abstract

Arabic is a highly diglossic language where most daily communication occurs in regional dialects rather than Modern Standard Arabic (MSA). Despite this, machine translation (MT) systems often generalize poorly to dialectal input, limiting their utility for millions of speakers. We introduce Alexandria, a large-scale, community-driven, human-translated dataset designed to bridge this gap. Alexandria covers 13 Arab countries and 11 high-impact domains, including health, education, and agriculture. Unlike previous resources, Alexandria provides unprecedented granularity by associating contributions with city-of-origin metadata, capturing authentic local varieties beyond coarse regional labels. The dataset consists of parallel English-Dialectal Arabic multi-turn conversational scenarios annotated with speaker-addressee gender configurations, enabling the study of gender-conditioned variation in dialectal use. Comprising 107K total turns, Alexandria serves as both a training resource and as a rigorous benchmark for evaluating MT and Large Language Models (LLMs). Our automatic and human evaluation benchmarks the current capabilities of Arabic-aware LLMs in translating across diverse Arabic dialects and sub-dialects while exposing significant persistent challenges. The Alexandria dataset, the creation prompts, the translation and revision guidelines, and the evaluation code are publicly available in the following repository: https://github.com/UBC-NLP/Alexandria