LAILA: A Large Trait-Based Dataset for Arabic Automated Essay Scoring

AI summary

Overview

Research area: Natural Language Processing, specifically Automated Essay Scoring (AES) for Arabic, with a focus on dataset construction and benchmarking of scoring models.

Technical level: Intermediate. The dataset design and annotation procedure are accessible to a general reader, while the benchmarking section assumes familiarity with model families (feature-based, encoder-based, large language models) and evaluation metrics such as Quadratic Weighted Kappa (QWK).

Scope in one sentence: The paper introduces LAILA, described as the largest publicly available Arabic AES dataset to date, comprising 7,859 essays across 8 prompts with holistic and trait-specific annotations on seven writing dimensions, and reports baseline scoring results for prompt-specific and cross-prompt settings.

What This Paper Is About

Automated Essay Scoring research has advanced far more for English than for Arabic, largely because English has large public annotated datasets while Arabic resources remain small, lack trait-level annotations, or are not publicly released. The authors build and publicly release LAILA to close this gap, then benchmark state-of-the-art Arabic and English models on it to establish reproducible baselines.

Key Contributions

1. Novel large-scale dataset: The paper introduces the first large-scale dataset of Arabic essays for AES, comprising 7,859 essays across 8 prompts (5 persuasive, 3 explanatory) written by 4,372 students from 24 high schools in Qatar in grades 10 to 12.
2. Annotation guidelines: The authors develop and share comprehensive annotation guidelines and a guidebook to ensure transparency and reproducibility of the scoring process.
3. Trait-specific annotations: Each essay carries a holistic score plus scores on seven traits (relevance, organization, vocabulary, style, development, mechanics, grammar), with inter-annotator agreement reported per trait and per prompt.
4. Benchmarking and public release: Baselines are established under prompt-specific and cross-prompt evaluation setups using feature-based models, encoder-based models, and LLMs, and both the dataset, the predefined splits, and the benchmarking code are released publicly.

Main Findings

• Dataset scale and composition: LAILA contains 7,859 essays, with 3,446 explanatory and 4,413 persuasive. Essays per prompt range from 500 to 1,181, average length is 171 words, and maximum length is 706 words.
• Score distributions: Most traits follow a near-normal pattern on the 6-point scale. 86% of essays agree strongly with the prompt on relevance (REL). Grammar (GRA) and Vocabulary (VOC) show the highest percentage of score 1 at 16%, followed by Mechanics (MEC) at 13%. Extreme scores are rare, with Style (STY) having the highest share of score 5 at 6%.
• Inter-annotator agreement: Measured with Quadratic Weighted Kappa between the two main annotators, average agreement per prompt ranged from 0.66 (P5) to 0.75 (P1), described as substantial. Organization (ORG) showed the strongest agreement, while REL, Development (DEV), and MEC had lower agreement. P5 had the highest share of essays escalated to a third annotator at 23.3%.
• Prompt-specific best model: AraBERT achieved the best overall prompt-specific performance with an average QWK of 0.74, leading in 7 of 8 traits, while AraT5 trailed by 1.6 points and led on REL.
• Prompt-specific LLM gap: The best LLM variant, Fanar with 5-shot prompting, scored 16.5 points below AraBERT in the prompt-specific setup, although few-shot examples improved every LLM.
• Cross-prompt best model: MOOSE achieved the highest average cross-prompt performance with an average QWK of 0.597, followed closely by the feature-based XGB at 0.593 and AraBERT at 0.549. ProTACT showed the lowest performance overall among encoder-based models.
• Cross-prompt trait leaders: No single model dominated all traits. MOOSE performed best in REL, STY, and GRA; RF led in VOC and HOL; XGB in ORG; and Fanar with 5-shot prompting in DEV and MEC.
• Cross-prompt LLM sensitivity: Few-shot prompting improved Fanar and R7B by 19 and 11.6 points respectively, whereas ALLaM dropped by 7 points, indicating high sensitivity to prompt design.
• Setup gap: Encoder-based models excel in the prompt-specific setup, approaching human agreement levels, but their performance drops sharply in the cross-prompt setup, suggesting they capture prompt-dependent patterns more than generalizable features.

Methodology in Plain English

The authors designed LAILA around nine stated design principles covering authenticity, diversity, gender balance, full trait coverage, exam-style writing conditions, participant privacy, standardized annotation, scalability and reusability, and support for multiple evaluation setups.

Collection. With Institutional Review Board approval (IRB Number: QU-IRB 159/2024-EA), the team visited 24 high schools in Qatar over one academic year, conducting 27 visits (24 initial, three follow-ups) by nine team members, each lasting around six hours. Students from grades 10 to 12 wrote essays under timed conditions using a Microsoft Form with a 65-minute limit, without access to external materials. Approximately 30% of participants were low-performing, 40% medium-performing, and 30% high-performing. Participation was voluntary and required parental and student consent.

Cleaning. Duplicates and misidentified submissions were resolved through manual verification and name cross-referencing, submissions lacking meaningful content were removed, and essays with 10 or fewer words were excluded.

Annotation. A team of 6 annotators and 3 supervisors, all Arabic language teachers or lecturers, scored the essays on the Core Academic Skills Test (CAST) rubric from the Qatar University Testing Center. Six traits were rated 0 to 5, relevance was rated 0 to 2, and holistic score was the sum of the seven traits. If relevance was 0, all other traits were set to 0. Each essay was independently scored by two annotators. If the holistic scores differed by less than 6 points, the mean was computed and rounded down; if the difference was 6 points or more, a supervisor adjudicated and provided the final score.

Benchmarking. Models were compared in three categories: feature-based models (Linear Regression, Random Forest, XGBoost, a feedforward Neural Network), encoder-based models (AraBERT with handcrafted features, AraT5, and for cross-prompt ProTACT and MOOSE), and LLMs (ALLaM, Command-R7B-Arabic, and Fanar) in zero-shot and 5-shot settings. All models used a multi-task setup predicting holistic and trait scores jointly, leveraging 816 handcrafted features. Hyperparameters were tuned with Bayesian optimization using Optuna's TPESampler, with 20 trials, 5 startup trials, and a fixed random seed of 11. The prompt-specific setup used 5-fold cross-validation, and the cross-prompt setup used 8-fold leave-one-prompt-out cross-validation with 2 prompts for development and 5 for training.

Why This Matters

Impact on research. Arabic AES research has been constrained by fragmented, small resources typically under 2,000 essays, with several existing Arabic datasets either proprietary or lacking trait-level annotations. LAILA provides a large, public, richly annotated benchmark that is comparable in scale and annotation depth to leading datasets in other languages, enabling standardized comparison and reproducible model development.

Real-world applications:

• Automated grading and scoring of Arabic student essays in classroom and exam settings.
• Timely, detailed feedback tools for Arabic writing instruction across multiple writing traits.
• Educational assessment and testing centers building Arabic writing evaluation pipelines.
• Adaptive learning platforms that need to diagnose specific writing weaknesses in Arabic.

Industry relevance. Educational technology companies, testing organizations, and ministries of education working in Arabic-language contexts can use the dataset and released benchmarking code to build or evaluate scoring systems. The paper notes that while research on automated scoring of English essays began more than 55 years ago, Arabic lags behind, so a shared public benchmark lowers the barrier to entry for commercial and institutional systems.

Future Directions

• Prioritize cross-prompt AES: The authors state that cross-prompt scoring is closer to real-world conditions (where labeled essays for the target prompt are often unavailable), and that best models remain far from human performance there.
• Broaden genre coverage: LAILA only includes explanatory and persuasive prompts, so extending to narrative or descriptive writing would improve model robustness.
• Improve geographic and educational diversity: The data comes from a single country (Qatar) and grades 10 to 12; generalizing to other Arabic-speaking populations and other educational systems is an open question.
• Address the relevance scale and essay length: Relevance is scored only 0 to 2, which limits granularity and may cause QWK to underrepresent subtle differences, and the average essay length of 171 words limits assessment on long-form academic tasks.

Target Audience

Arabic NLP researchers, automated essay scoring researchers, and educational assessment specialists who need a benchmark dataset for Arabic writing evaluation. It is also useful for Arabic language educators and curriculum designers, and for data scientists and engineers in educational technology who want to build or evaluate Arabic essay scoring systems, or who want to study the strengths and weaknesses of feature-based, encoder-based, and LLM approaches under both prompt-specific and cross-prompt conditions.

Authors’ abstract

Automated Essay Scoring (AES) has gained increasing attention in recent years, yet research on Arabic AES remains limited due to the lack of publicly available datasets. To address this, we introduce LAILA, the largest publicly available Arabic AES dataset to date, comprising 7,859 essays annotated with holistic and trait-specific scores on seven dimensions: relevance, organization, vocabulary, style, development, mechanics, and grammar. We detail the dataset design, collection, and annotations, and provide benchmark results using state-of-the-art Arabic and English models in prompt-specific and cross-prompt settings. LAILA fills a critical need in Arabic AES research, supporting the development of robust scoring systems.