EduAdapt: A Question Answer Benchmark Dataset for Evaluating Grade-Level Adaptability in LLMs

AI summary

Overview

Research area: Natural Language Processing / Educational AI — benchmark design and evaluation of large language models for K-12 science question answering.

Technical level: Intermediate. The methods are conceptually approachable (data generation, prompting, evaluation), but familiarity with LLM benchmarking, prompting pipelines, and metrics like accuracy and LLM-as-a-judge will help.

Scope: The paper introduces EduAdapt, a ~48,000-item grade-labeled question-answer benchmark spanning Grades 1-12 and nine science subjects, and uses it to show that current open-source LLMs fail to adapt their responses to younger students' developmental levels.

What This Paper Is About

Large language models can answer academic questions at a high level, but they rarely adjust vocabulary, explanation depth, and tone to match a student's grade. This matters most in K-12 education, where age-appropriate language is essential for learning. The authors build the first standardized benchmark that measures how well LLMs adapt their answers across the full K-12 grade range, and use it to expose where current models break down.

Key Contributions

1. EduAdapt benchmark dataset: 47,734 grade-labeled QA pairs (multiple-choice and open-ended) across Grades 1-12, covering nine science subjects (Biology, Physics, Chemistry, Computer Science, Ecology, Geography, Geology, Medicine, Meteorology), grouped into four developmental bands: Grades 1-2, 3-5, 6-8, and 9-12, aligned with Next Generation Science Standards (NGSS).

2. A generation-plus-verification pipeline: Wikipedia content is grade-classified, converted into QA pairs with grade-specific prompts, filtered by a self-reflection mechanism scoring each pair on five pedagogical criteria, then checked by human reviewers on a 1,000-item sample.

3. The first evaluation framework for grade-level adaptability: standardized MCQ accuracy and LLM-as-a-judge scoring protocols applied to nine open-source models of varying sizes and families, making cross-model and cross-grade comparisons possible.

4. A released train/dev/test split (28,640 / 9,547 / 9,547) that supports both benchmarking and fine-tuning toward grade-aware educational models.

Main Findings

• LLMs are far weaker on early-grade content. Small models (1.5B-3B) score only 50-60% on MCQs for Grades 1-5 but climb to 70-80% on Grades 9-12. Open-ended scores show the same pattern across nearly every model.

• Model size helps, but does not solve the problem. Qwen2.5-14B and Mistral-Small-24B outperform smaller models on every grade level yet still register their weakest results on Grades 1-5. Larger models get better, not developmentally smarter.

• Qwen models lead at matched sizes. Qwen2.5-14B performs on par with the much larger Mistral-Small-24B on MCQs, and the Qwen family consistently outperforms same-size competitors. One notable outlier: Qwen2.5-7B scored 92.0% on Grade 1-2 MCQs, beating every larger model there.

• Even constrained MCQs remain hard. Despite being the easier format, MCQs produced widespread underperformance in smaller models, indicating that the dataset captures genuine grade-specific knowledge gaps rather than just generation difficulty.

• Standard text-overlap metrics fail for this task. BLEU, ROUGE, and BERTScore could not distinguish correct from incorrect answers; BERTScore exceeded 90% even for semantically wrong or developmentally inappropriate responses. Accuracy and LLM-as-a-judge scoring were far more informative.

• Human evaluation confirms dataset quality with strong agreement. Three independent reviewers gave average scores from 7.69 (Grades 1-2) to 8.86 (Grades 9-12), with Fleiss' Kappa rising from 0.668 to 0.860 across the same range.

• Error patterns are consistent and grade-dependent. Gemma-2B was fluent but hallucination-prone; LLaMA3-8B was lively and child-friendly but factually imprecise; Mistral-Small-24B was readable and well-structured but suffered from hallucination, incomplete answers, and conceptual misalignment. Across all three, readability was a strength while correctness and grade alignment were the recurring failures.

Methodology in Plain English

The authors started with cleaned Wikipedia articles from nine science domains. Because Wikipedia is written at a high reading level, they used Phi-4 (a 14B model tuned for educational reasoning) to tag each passage as suited to Grades 1-2, 3-5, 6-8, or 9-12; roughly 14% of passages landed in the early-grade range.

Next, they wrote separate generation prompts for each grade band, specifying sentence length, vocabulary, and whether the question should ask for observation, cause-and-effect reasoning, or abstract modeling. This produced about 166,000 QA pairs.

To raise quality, they applied a self-reflection step: the same model re-read each pair and rated it 1-10 on five criteria — language appropriateness, grade alignment, relevance, clarity, and subject fit. Any pair scoring below 8 on even one criterion was discarded, leaving 47,734 items.

Then came human verification. Three education-focused contractors reviewed a 1,000-item sample (about 10% of the test set) using the same five criteria. Their agreement was measured with Fleiss' Kappa.

For evaluation, nine open-source instruct-tuned models (Qwen2.5-1.5B/3B/7B/14B, SmolLM-1.7B, Gemma-2B-it, LLaMA3.2-3B, LLaMA3-8B, Mistral-Small-24B) were given grade-specific prompts. MCQs were scored by accuracy; open-ended answers were graded by three separate LLM judges (GPT-4o, Qwen2.5-72B, LLaMA3.3-70B) on a 1-10 scale against reference answers. All inference ran on NVIDIA RTX A6000 GPUs with vLLM.

Why This Matters

Impact on research: This is the first benchmark that treats grade-level adaptability as a first-class evaluation target rather than a side effect of general capability. It gives the educational NLP community a shared yardstick and shows that scaling model parameters is not the same as scaling pedagogical appropriateness — a finding with direct implications for how educational models should be trained and prompted.

Real-world applications:

• K-12 tutoring systems that need to explain the same concept differently to a 7-year-old and a 17-year-old.
• Curriculum-aligned content generation for teachers producing worksheets, quizzes, and study guides at specific grade levels.
• Safety and appropriateness screening of AI responses delivered to children, where overly advanced or vague answers are a real risk given that children spend over five hours daily on screens.
• Model selection and fine-tuning — schools and edtech vendors can use EduAdapt scores to choose or adapt models for specific grade bands instead of relying on general academic benchmarks.

Industry relevance: Edtech companies, publishers, and platform builders deploying AI to student audiences gain a concrete way to audit grade alignment before release. The failure of BLEU/ROUGE/BERTScore also matters operationally: teams using these metrics as quality gates may be shipping developmentally inappropriate content without detecting it.

Future Directions

1. Fix the grade imbalance. The dataset has far fewer QA pairs for Grades 1-5 than for Grades 6-12, which limits the robustness of early-grade conclusions. Balanced generation is the obvious next step.

2. Extend beyond a single curriculum. The benchmark follows a K-12 framework that is largely US-centric; adapting it for other national curricula, cognitive expectations, and regional contexts would broaden its validity.

3. Add multimodal and multilingual coverage. Diagram-based and image-based questions better reflect real assessments, and non-English QA would serve far more students.

4. Close the early-grade gap through training. The paper points to data augmentation, curriculum-aligned pretraining, and targeted fine-tuning as the interventions most likely to help models handle younger learners — an open empirical question the benchmark is now positioned to measure.

Target Audience

This paper is most valuable to researchers building educational NLP systems and benchmarks, edtech engineers selecting or fine-tuning models for student-facing products, curriculum designers and learning scientists interested in age-appropriate AI content, and evaluation researchers who need a concrete case study on why surface-level text metrics fail for pedagogically grounded tasks.

Authors’ abstract

Large language models (LLMs) are transforming education by answering questions, explaining complex concepts, and generating content across a wide range of subjects. Despite strong performance on academic benchmarks, they often fail to tailor responses to students' grade levels. This is a critical need in K-12 education, where age-appropriate vocabulary and explanation are essential for effective learning. Existing models frequently produce outputs that are too advanced or vague for younger learners, and there are no standardized benchmarks to evaluate their ability to adjust across cognitive and developmental stages. To address this gap, we introduce EduAdapt, a benchmark of nearly 48k grade-labeled QA pairs across nine science subjects, spanning Grades 1-12 and grouped into four grade levels. We evaluate a diverse set of open-source LLMs on EduAdapt and find that while larger models generally perform better, they still struggle with generating suitable responses for early-grade students (Grades 1-5). Our work presents the first dataset and evaluation framework for assessing grade-level adaptability in LLMs, aiming to foster more developmentally aligned educational AI systems through better training and prompting strategies. EduAdapt code and datasets are publicly available at https://github.com/NaumanNaeem/EduAdapt.