AraLingBench A Human-Annotated Benchmark for Evaluating Arabic Linguistic Capabilities of Large Language Models

AI summary

Overview

Research area: Arabic natural language processing; large language model evaluation and benchmarking.

Technical level: Intermediate. The paper reports a benchmark construction process and empirical results rather than new model architectures, so it is accessible to readers familiar with basic LLM evaluation concepts.

Scope: AraLingBench is a 150-question, fully human-annotated Arabic multiple-choice benchmark that isolates five core linguistic skills (grammar, morphology, spelling, reading comprehension, syntax) and is used to evaluate 35 Arabic and bilingual LLMs from 350M to 70B parameters.

What This Paper Is About

Existing Arabic benchmarks such as ArabicMMLU, EXAMS, BALSAM, CamelEval, and 3LM mostly test factual recall, academic knowledge, or task performance, and they treat linguistic competence as an unmeasured prerequisite. The authors argue that no existing benchmark directly tests whether a model handles Arabic grammatical agreement, morphological derivation, or orthographic conventions. AraLingBench is built to fill that gap by making fundamental Arabic linguistic understanding the explicit target of evaluation rather than a byproduct of knowledge testing.

Key Contributions

1. A new linguistically focused benchmark. The authors introduce AraLingBench, a fully human-annotated benchmark of 150 multiple-choice questions distributed evenly across five categories (30 per category): grammar, morphology, spelling, reading comprehension, and syntax. All items were authored and reviewed by Arabic linguistics experts.

2. A large-scale evaluation of Arabic LLMs. They evaluate 35 Arabic and bilingual models, ranging from 350M to 70B parameters and drawn from the Open Arabic LLM Leaderboard, using zero-shot prompting with no few-shot examples or chain-of-thought.

3. Evidence that general benchmark scores do not predict linguistic competence. Cross-benchmark analysis across seven other Arabic benchmarks shows strong correlations with some evaluations (ArabicMMLU at r = 0.884, EXAMS at r = 0.784, AraTrust at r = 0.751) but a negative correlation with retrieval-augmented evaluation (ALRAGE at r = -0.539).

4. A diagnostic analysis of skill structure and difficulty. The paper reports inter-category correlations, per-category performance tiers, and human-annotated difficulty labels to show which linguistic skills cluster together and how human-perceived difficulty diverges from model performance.

Main Findings

• Performance stratifies into tiers. Top performers (Yehia-7B, ALLaM-7B) reach 72–74% average accuracy; mid-tier models (Fanar, Qwen2.5-14B variants) reach 55–62%; smaller or less specialized models remain below 50%.

• Syntax is the hardest category. Syntax is the most challenging category with a median of approximately 48%, while Spelling and Reading Comprehension are the easiest with medians of approximately 58–60%. Even the best models only reach 60.0% on Syntax.

• Strong models are uneven across skills. Yehia-7B attains 86.7% on Spelling but only 53.3% on Syntax, a 33-point gap. Interquartile ranges of 15–20 percentage points indicate substantial heterogeneity in category-level performance across architectures and training regimes.

• Morphology remains a persistent weakness. Models consistently underperform in morphology (median approximately 60%), and a top performer such as Yehia reaches only 80% despite Arabic's morphological richness.

• Linguistic skills form overlapping clusters, not a single construct. Grammar and morphology correlate highly (r = 0.83 in the text; the figure caption reports r = 0.80), as do spelling and grammar (r = 0.86). Spelling and reading comprehension correlate moderately (r ≈ 0.51). Syntax shows the weakest correlations with other categories (r ≈ 0.13–0.40), suggesting it relies on distinct mechanisms.

• Knowledge benchmarks do not transfer reliably. Hala-9B attains 65.6% on ArabicMMLU but only 54.7% on AraLingBench, and the top performers on ArabicMMLU are not necessarily the top performers on AraLingBench.

• Training regime matters. Models heavily tuned on synthetic instruction data (for example, the Hala family) score highly on knowledge benchmarks but underperform on linguistic ones. Models fine-tuned with real instruction data (Yehia, ALLaM) show stronger alignment between general and linguistic competence.

• Difficulty does not scale monotonically with model accuracy. Median accuracy is 58% on Easy questions, 50% on Medium, and 54% on Hard. Qwen3-8B-Base scores 73.1% on Hard items versus 50.0% on Medium and 58.0% on Easy. Leading models degrade modestly from 76% to 69% across levels, and difficulty-level correlations are strong (r > 0.65), meaning model rankings stay consistent even when accuracy patterns do not.

• Benchmark composition details. Difficulty labels are 50 Easy (33.3%), 74 Medium (49.3%), and 26 Hard (17.3%). Format is 125 four-choice questions (83.3%) and 25 three-option questions (16.7%). Correct answers appear at position A 34.0%, B 27.3%, C 26.0%, and D 12.7% of the time.

Methodology in Plain English

The authors used a four-phase expert-driven construction process. First, five Arabic linguistics experts from the American University of Beirut, all with advanced training in Arabic grammar, morphology, and syntax, authored original question–answer pairs covering the five categories. They were allowed to consult textbooks and exam archives for inspiration but had to compose novel questions.

Second, a separate group of native Arabic speakers who were not experts reviewed the questions for clarity and perceived difficulty. Questions were kept only if they were sufficiently challenging for the validation group and covered a diverse range of linguistic phenomena and formats.

Third, a senior Arabic linguist reviewed every candidate item for accuracy, phrasing, and category alignment. Items were refined to have unambiguous wording and exactly one correct answer, and questions that could fit multiple categories were assigned to the category matching their primary linguistic focus.

Fourth, three independent annotators rated each question on a three-point scale (1, 2, 3) for Easy, Medium, and Hard, with the final label determined by majority vote.

The result was 150 validated questions, 30 per category. For evaluation, every model received each question fully in Arabic with options labeled A–D and was asked for a single-letter answer, which was matched automatically against the gold label. All testing was zero-shot, with no few-shot examples or chain-of-thought prompting. Accuracy per category and average accuracy across the five categories were the reported metrics. The authors then examined per-category results, correlations between categories, correlations between AraLingBench and seven other Arabic benchmarks, and performance across difficulty levels. The evaluation code is publicly available on GitHub.

Why This Matters

Impact on research. The paper argues that the Arabic evaluation ecosystem, with over 40 distinct benchmarks according to cited work, is dominated by knowledge-intensive tests. AraLingBench repositions linguistic competence as a measurable target rather than an assumed prerequisite, giving researchers a way to distinguish models that produce fluent Arabic from those that internalize its grammatical and morphological structure. The negative correlation with retrieval-augmented evaluation (ALRAGE, r = -0.539) suggests that some optimization pathways can inflate apparent performance without improving underlying understanding.

Real-world applications:

• Model selection for Arabic deployments. Organizations can use category-level results to pick models whose strengths match their use case, rather than relying on aggregate knowledge-benchmark rankings.
• Educational technology. Grammar, morphology, and spelling diagnostics are directly relevant to Arabic language learning tools and automated assessment of student writing.
• Content generation and publishing. Reliable orthography and grammatical agreement matter for Arabic media, localization, and editorial workflows.
• Model development and fine-tuning. The benchmark identifies specific weaknesses (syntax and morphology) that can guide targeted data curation and training interventions for Arabic models.

Industry relevance. The evaluation covers models from 350M to 70B parameters, including small models suitable for on-device or low-cost deployment, so the tiering results are relevant to practitioners with different compute budgets. The finding that synthetic instruction data can raise knowledge-benchmark scores without improving linguistic competence is a practical caution for teams building Arabic products on fine-tuned or retrieval-augmented pipelines.

Future Directions

• Targeted work on syntax and morphology. Syntax shows the weakest correlations with all other categories and a ceiling of 60.0% even among top models, raising the question of whether architectural inductive biases such as hierarchical attention or structural modeling are needed.

• Better difficulty calibration. Human-annotated difficulty does not align with model accuracy (58% Easy, 50% Medium, 54% Hard), so the authors call for calibrating benchmark difficulty using both human annotation and pilot testing on representative models.

• Expanding and extending the benchmark. The paper does not report a plan for enlarging the 150-question set, adding categories, or covering dialectal variation, though its design as a complement to existing suites invites extension.

• Separating memorization from comprehension diagnosis. Since models appear to succeed through memorization or pattern recognition, an open question is how to design items and analyses that reliably isolate genuine linguistic understanding from retrieval-based shortcuts.

Target Audience

This paper is most useful to Arabic NLP researchers and benchmark designers, engineers building or fine-tuning Arabic LLMs, evaluation and model-selection teams comparing Arabic models, and linguists interested in how computational systems handle Arabic grammar, morphology, orthography, and syntax. Readers looking for a benchmark they can run directly will find the evaluation code on GitHub, while readers interested in evaluation methodology will find the cross-benchmark correlation analysis and difficulty findings the most distinctive parts.

Authors’ abstract

We present AraLingBench: a fully human annotated benchmark for evaluating the Arabic linguistic competence of large language models (LLMs). The benchmark spans five core categories: grammar, morphology, spelling, reading comprehension, and syntax, through 150 expert-designed multiple choice questions that directly assess structural language understanding. Evaluating 35 Arabic and bilingual LLMs reveals that current models demonstrate strong surface level proficiency but struggle with deeper grammatical and syntactic reasoning. AraLingBench highlights a persistent gap between high scores on knowledge-based benchmarks and true linguistic mastery, showing that many models succeed through memorization or pattern recognition rather than authentic comprehension. By isolating and measuring fundamental linguistic skills, AraLingBench provides a diagnostic framework for developing Arabic LLMs. The full evaluation code is publicly available on GitHub.