QuranicMMLU: A Cognitively-Aware Benchmark for Evaluating Generative AI Solutions on Quranic Linguistic Knowledge

AI summary

Overview

Research area: Natural Language Processing, specifically benchmark construction and evaluation for Arabic and Islamic-domain large language models.

Technical level: Intermediate.

Scope: The paper builds and releases QuranicMMLU, a 980-question human-reviewed benchmark that evaluates generative AI on Quranic Arabic across five linguistic pillars, five Bloom's cognitive levels, and three verse-perplexity strata, in both multiple-choice and open-ended formats.

What This Paper Is About

Existing Quranic and Islamic benchmarks test general question answering, factual recall, and semantic retrieval, but they do not probe specific linguistic competencies in Quranic Arabic, and they rarely measure how cognitively demanding a question is. Most are also multiple-choice only, a format that the paper argues can be gamed through answer-choice artifacts and that differs from how users actually query models. The goal of QuranicMMLU is to provide a fine-grained, linguistically grounded evaluation grid that separates where models genuinely understand Quranic Arabic from where aggregate accuracy hides failure.

Key Contributions

1. A five-pillar taxonomy of Quranic Arabic covering Phonology, Morphology, Syntax, Semantics, and Pragmatics, organized as 5 pillars, 12 categories, and 31 leaves, with leaves ranging from tajwīd and root-and-pattern morphology to occasions of revelation and inter-surah coherence.
2. A human-reviewed, retrieval-grounded dataset of 980 questions, each issued in both open-ended and multiple-choice form, stratified by Bloom's cognitive level (Remember, Understand, Apply, Analyze, Evaluate) and by verse perplexity, computed using the Gemma-3-12B model.
3. An evaluation of 12 systems on the identical item pool in both formats, showing that multiple-choice scoring overstates ability and hides failures that appear only once answer choices are removed.
4. A public leaderboard and resources released at the Hugging Face Space musiml-org/QuranicMMLU.

Main Findings

• Specialized model leads: Ansari, a system specialized for Islamic and Quranic content, ranked first in both formats, with 96.4% multiple-choice accuracy and a 93.2% open-ended Gemini judge score.
• Format gap is systematic: Every one of the 12 systems scored higher on multiple-choice accuracy (average 84%) than on open-ended answer quality (average 60%). On multiple-choice, the seven highest-scoring systems each cleared 84%, and the weakest system, ALLaM-7B, still reached 69.5% against a 25% chance level.
• Rankings largely agree: The two orderings are closely aligned (Kendall's tau = +0.7273; 57 of 66 model pairs concordant), with Ansari and gemma-4-31B holding the top two positions in both.
• Format-sensitive outlier: Fanar-Sadiq moved up four places under the open-ended setting (eighth in multiple-choice, fourth in open-ended) not because it improved in absolute terms, but because other systems degraded more sharply when answer choices were removed.
• Phonology is the weakest pillar: Phonology yielded the weakest scores for almost every system in both settings, dropping below 34% for most systems on open-ended questions, while Morphology and Semantics were consistently the strongest.
• Bloom-level profile: Scores peaked at the Understand level and fell to their lowest at Apply and Analyze in both settings, suggesting systems recognize and explain Quranic phenomena more readily than they apply rules to new cases.
• Threshold performance: Ansari scored 80% or higher on 91.1% of open-ended items; gemma-4-31B averaged 82.5% and cleared that threshold on 75.6% of items.
• Review was consequential: Of 1,086 candidates, 370 (34%) were revised, most often a corrected Bloom level (235 items); 106 (10%) were discarded; 610 were accepted as generated, leaving the final 980. Conflicts were settled by majority verdict among three adjudicators, with 133 items escalated to group discussion.
• Judge agreement: On the multiple-choice answer, the three independently prompted models (the generator and the two judges) showed almost-perfect agreement (Fleiss's kappa = 0.90).

Methodology in Plain English

The team built the benchmark in six stages.

First, they defined a taxonomy by studying the literature and key sources for each of the five linguistic pillars, drafting candidate categories, refining them with the help of AI deep-research tools, and grouping related items. This produced 5 pillars, 12 categories, and 31 leaves.

Second, they generated questions. For each leaf, Claude Opus 4.7 produced questions, reference short answers, and multiple-choice options at each Bloom level. Generation started by finding a verse containing an instance of the target phenomenon, drawn from grounding datasets, and manually confirmed. Verses were stratified into low, medium, and high perplexity using Gemma-3-12B as a proxy for model familiarity, applied to single-verse questions rather than whole-surah ones. Each question used a four-layer prompt stack, with a specialized prompt per leaf and Bloom level. Every item was written to support both open-ended and multiple-choice answering, with four options per question: one correct, two close distractors, and one clearly wrong.

Third, two LLM judges, GPT-5.2 and Gemini-2.5-Flash, annotated each item. Each judge wrote an open-ended answer, selected the correct multiple-choice option, assigned a Bloom level, and scored the question out of five on specificity, answerability, leaf relevance, and clarity. Both judges had full access to the grounding datasets, making annotation retrieval-augmented rather than memory-dependent.

Fourth, every annotation was reviewed by two humans per question — undergraduate students, fluent Arabic speakers with a background in Islamic knowledge. Because they are not religious scholars, decisions were grounded in corpus evidence and cited references following a written guide that ranks sources of truth: corpus evidence first, then established scholarly consensus, then genuine dispute. Reviewers could edit questions, correct Bloom levels, choose among annotated answers, add a new answer, or delete items. Any item with a suspected hallucination, an unresolved scholarly dispute, or a single flagged concern was discarded when it could not be cleanly fixed.

Fifth, the final 980 items were benchmarked in both formats against 12 open-weight and open-access systems, including general-purpose models (Gemma, Qwen/Qwen-VL, and the Arabic-centric ALLaM) and Islamic-specialized models (Fanar-Sadiq and Ansari).

Sixth, open-ended answers were scored 0 to 5 by Gemini as an automated grader given the question, a reference answer, and the model's response. The paper reports this as a percentage of the 5-point maximum, along with the proportion of responses scoring 80% or higher. Multiple-choice is scored as accuracy. Kendall's tau compares the two rankings.

Grounding sources were the Quran.com API for Phonology; the Quranic Arabic Corpus for Morphology and Syntax; Quranic tafsīr from Tarteel for Semantics; and an asbāb al-nuzūl dataset based on Sahih Asbab al-Nuzul (Ibrahim Muhammad al-Ali) for Pragmatics.

Why This Matters

The paper addresses a domain where the cost of ungrounded answers is unusually high: the Quran is the primary religious and cultural text for nearly two billion Muslims, and hallucinated or inaccurate responses carry real religious consequences. By separating linguistic competencies and cognitive demand, the benchmark shows that a single accuracy number can conceal which abilities a model actually has.

Real-world applications:

• Selecting and auditing models for Islamic question-answering assistants, where multiple-choice performance alone would overstate readiness for free-form user queries.
• Developing and evaluating Arabic NLP tooling more broadly, since the pillars target phenomena such as root-and-pattern morphology and undiacritized-script disambiguation that general Arabic benchmarks do not isolate.
• Building educational or recitation-support tools, since Phonology — covering madd, qalqalah, waqf, and qirā'āt — is the pillar where nearly every system performed worst.
• Informing theological and scholarly review workflows, given the paper's explicit caution that the dataset is for benchmarking model capabilities and should not be treated as an authoritative source of religious fact.

Industry relevance: The results give model developers a per-pillar and per-Bloom diagnostic rather than an aggregate score, and they show that domain specialization (Ansari, Fanar-Sadiq) changes measured behavior in ways that general-purpose scaling does not straightforwardly replicate — the largest model evaluated, Qwen3-235B-A22B, sat in the lower half of the open-ended ranking at 52.3%.

Future Directions

• Broadening leaf coverage, since the authors state the taxonomy does not capture every linguistic rule or concept within its five pillars.
• Adding human scoring of open-ended answers to complement the LLM judge, which the authors note may introduce model bias, including a possible preference for answers resembling the judge's own model family.
• Evaluating further Islamic-specialized systems, including some top-ranked models on Islamic-domain leaderboards that remain unassessed.
• Conducting a dedicated error analysis of the LLM-as-judge scores to confirm whether multiple-choice benchmarks can serve as an efficient proxy for open-ended quality.
• Assessing closed-source commercial models, which were excluded because the authors did not have API access, and mitigating single-generator artifacts, since Claude Opus 4.7 generated all questions.

Target Audience

Researchers and practitioners in Arabic NLP and Islamic-domain AI who need to evaluate generative models on Quranic Arabic; benchmark designers interested in cognitive stratification and format-aware evaluation; developers of Islamic-specialized and Arabic-centric language models; and educators or institutions assessing whether a model is reliable enough for Quranic content. The paper is written for readers comfortable with benchmark methodology and metric interpretation, though the taxonomy and results are presented clearly enough for a non-specialist technical reader. Some background in Arabic linguistics helps with the pillar descriptions but is not required to follow the findings.

Authors’ abstract

We introduce QuranicMMLU, a benchmark for evaluating generative AI on Quranic Arabic across multiple dimensions of linguistic complexity. Existing Quranic benchmarks center on general question answering and semantic retrieval, without probing specific linguistic competencies or stratifying by cognitive demand and verse difficulty. We construct a five-pillar Quranic taxonomy spanning Phonology, Morphology, Syntax, Semantics, and Pragmatics, with 31 leaves covering phenomena from tajwīd and root-and-pattern morphology to occasions of revelation and inter-surah coherence. For each leaf we generate questions stratified by Bloom's cognitive level and verse perplexity, then have LLM as a judge to independently answer and score every item and route the annotations to manual review. The resulting dataset comprises 980 human-reviewed questions, each issued in both open-ended and multiple-choice form. We benchmark 12 systems on these items and find that the Islamic-specialized model leads, yet every system scores higher on multiple-choice accuracy (average 84%) than open-ended answer quality (average 60%): the two rankings agree closely (Kendall's τ=0.73), but multiple-choice scoring hides failures that surface only once answer choices are removed. QuranicMMLU thus offers a rigorous, linguistically grounded framework for evaluating Arabic NLP in the Quranic domain.