YallaMorph: A Benchmark for Evaluating Arabic Morphological Generation in Large Language Models

AI summary

Overview

Research area: Natural Language Processing — Arabic computational morphology and LLM evaluation.

Technical level: Intermediate. The benchmark design and results are accessible to a general NLP audience, but the linguistic framing (root classes, stem allomorphy, clitic slots) assumes some familiarity with morphologically rich languages.

Scope: A large-scale, linguistically balanced benchmark of over 663K controlled Arabic morphological generation instances, used to test whether modern multilingual and Arabic-oriented LLMs can produce correct inflected and cliticized Arabic word forms from explicit lexical and feature specifications.

What This Paper Is About

Arabic is morphologically rich and complex: a single lemma can surface in thousands of forms through inflection plus cliticization (conjunctions, prepositions, articles, and pronouns), and the interactions involve templatic patterns and orthographic alternations that are easy to get subtly wrong. Existing Arabic evaluations mostly measure downstream task quality or overall fluency, which does not reveal whether a model can systematically realize a requested morphological configuration, such as "active imperfective, third-person feminine plural, with the object clitic -hu." The paper introduces YallaMorph, a benchmark that directly tests this controlled generation ability, and reports that even the strongest current LLMs fail on a substantial share of cases.

Key Contributions

1. A large-scale controlled benchmark for Arabic morphological generation. YallaMorph contains 663,804 entries covering verbs (perfective, imperfective, imperative, active and passive), nouns, and adjectives in both baseword (inflectional) and cliticized forms, derived from the CamelMorph MSA lexicon via CAMeL Tools.

2. A linguistically motivated sampling framework. Rather than exhaustively enumerating the (astronomically large) space of possible forms — estimated at over 4.4 billion with clitics — the benchmark samples lemmas along controlled strata: lemma frequency, root class, paradigm completeness, and stem complexity.

3. Inclusion of invalid (null) configurations. Roughly 13% of entries have no valid target form because the lemma's paradigm cannot accommodate the requested features (for example, asking for the feminine of a masculine-only noun). The benchmark therefore tests whether models can recognize impossible requests rather than hallucinate a plausible-looking form.

4. A systematic evaluation of seven instructed LLMs under zero-shot and 10-shot prompting, in English and Arabic prompts, and under diacritized, undiacritized, and orthographically normalized scoring, with breakdowns by linguistic and distributional category.

Main Findings

• Controlled Arabic morphology remains hard for all tested models. The best system (GPT) reaches 51.9% Any Match Accuracy in the strict diacritized 10-shot setting, rising to 67.7% when diacritics and certain orthographic variants are ignored. Most Arabic-specialized open models score in the single digits or low teens on the strict metric.

• Arabic specialization does not guarantee morphological accuracy. The Arabic-oriented models (Fanar, Jais, ALLaM) substantially underperform the general multilingual models (GPT, Gemini). Fanar is the strongest of the Arabic-oriented group at 21.5% diacritized 10-shot AMA; ALLaM and Jais-8B are near the bottom at 4.7% and 3.4%.

• Few-shot prompting helps consistently. Every model improves from zero-shot to 10-shot prompting. For example, GPT rises from 46.2% to 51.9% diacritized AMA, and Fanar from 5.1% to 21.5%, showing that in-context examples matter a great deal for lower-performing models.

• English prompts generally outperform Arabic prompts. This holds for Fanar, Jais-70B, and ALLaM on most metrics, though Arabic prompts are better for ALLaM and Jais-8B on several zero-shot undiacritized and normalized measures.

• Cliticized, unseen, and morphologically rare forms are the hardest. The benchmark is deliberately long-tailed: 59.8% of target forms are unseen in the reference frequency list, and 12.9% are null references. Errors concentrate on clitic attachments, lemmas absent from training data, and rare root or stem classes.

• Multiple valid realizations exist but are rare. About 3.2% of entries have more than one gold form, so the evaluation uses an "any match" criterion for multi-form cases.

Methodology in Plain English

The researchers built their test set from an existing, well-documented Arabic morphological database (CamelMorph MSA), which supplies lemmas, part-of-speech tags, English glosses, and rich feature information. Instead of trying to list every possible Arabic word — which the paper estimates at roughly 4.4 billion forms once clitics are included — they sampled intelligently.

Each lemma was labeled along four dimensions: how frequent it is in a large Arabic corpus (low, medium, high), what kind of root it has (sound, geminated, hamzated, weak-initial, hollow, defective, or non-templatic), how complete its paradigm is (full, masculine-only, feminine-only), and what kind of stem complexity it exhibits (regular, defective-family, hamza-family, or several suffix/prefix-specific families). The authors then sampled up to 10 lemmas per combination of these labels for the inflectional subset, and up to 5 lemmas per combination for the cliticized subset, keeping the distribution linguistically balanced and interpretable.

The task itself is presented to a model as a slot-filling problem: given a lemma, its part of speech, a gloss, and a bundle of morphological features (aspect, person, gender, number, voice, mood for verbs; gender, number, case, state for nominals, plus proclitic and enclitic slots), the model must output the corresponding Arabic surface form — or explicitly say the configuration is invalid. Models return their answers in a fixed JSON schema, and predictions are compared against gold forms under three orthographic regimes. The primary metric is Any Match Accuracy, with precision, recall, and F1 also reported. The authors used position-independent, lightweight prompts with separate versions for verbs, nouns, and adjectives, and ran both zero-shot and 10-shot experiments.

Why This Matters

Impact on research. The paper separates fluency from morphological control, a distinction that broad Arabic generation benchmarks cannot make. It provides a reproducible, publicly available testbed (on GitHub under CAMeL-Lab) that future work can use to measure progress on a specific and well-defined capability, and its stratified sampling design lets researchers diagnose which morphological phenomena models handle and which they do not.

Real-world applications:

• Machine translation and localization — Arabic output that inflects correctly for gender, number, and attached pronouns is essential for natural-sounding translations, especially from pro-drop source languages.
• Educational and language-learning tools — a system that can generate and explain Arabic paradigms reliably could support vocabulary drills, grammar exercises, and learner feedback.
• Text-to-speech and diacritization pipelines — correct diacritized forms are needed for accurate pronunciation; the benchmark's strict diacritized setting directly targets this.
• Search and information retrieval — robust morphological generation supports query expansion and normalization across the many surface variants of a single lemma.

Industry relevance. Any organization deploying Arabic-language AI — translation services, customer support chatbots, content generation, government or media localization — depends on models that can inflect and cliticize Arabic correctly. The results show that current Arabic-specialized models are not automatically better at this, which has direct implications for model selection and for the value of targeted fine-tuning or tool augmentation.

Future Directions

1. Diagnosing and fixing specific failure modes. The paper shows cliticization and rare forms are the hardest categories; future work could build targeted training data or fine-tuning recipes for exactly those strata, and test whether gains transfer to held-out categories.

2. Extending coverage beyond Modern Standard Arabic. The benchmark is built on MSA only. Dialectal Arabic (Egyptian, Gulf, Levantine) and historical varieties have different paradigms and clitic inventories that would require new resources and careful sampling.

3. Comparing specialized morphological generators against LLMs. The paper evaluates LLMs but does not benchmark them head-to-head against finite-state or rule-based generators such as CamelMorph's own engine, which would clarify where neural approaches help and where deterministic tools remain superior.

4. Improving handling of null and multi-form outputs. Since 12.9% of entries are invalid configurations and 3.2% admit multiple valid realizations, further work could probe whether models can reliably detect impossibility and produce the full set of legitimate variants rather than a single guess.

Target Audience

This paper is most useful for NLP researchers working on Arabic, morphologically rich languages, or linguistic evaluation of LLMs; developers building Arabic language technologies who need to choose or fine-tune models; and computational morphologists interested in how well modern neural systems handle controlled, feature-specified generation compared to classical finite-state and lexicon-based approaches.

Authors’ abstract

Arabic morphology remains challenging for large language models, since fluent generation does not guarantee accurate morphosyntactic control. Existing Arabic evaluations mainly target downstream tasks and do not directly test controlled morphological generation from explicit lexical and feature-based input. We introduce YallaMorph, a large-scale benchmark for Arabic morphological generation covering verbs, nouns, adjectives, their cliticized forms, and invalid configurations. We evaluate multilingual and Arabic-oriented LLMs under diacritized and undiacritized settings over 600K benchmark entries. Results show that Arabic morphological generation remains difficult, especially for cliticized, unseen, and morphologically rare forms.