ELBench: A Multi-Dimensional Benchmark for Education-Facing Large Language Models

AI summary

Overview

• Research area: Natural Language Processing, specifically evaluation benchmarks for large language models deployed in educational settings.
• Technical level: Intermediate — the paper is readable without deep technical background, though it assumes familiarity with benchmark terminology (accuracy scores, confidence intervals, correlation coefficients, LLM-as-judge).
• Scope: This paper introduces ELBench, a four-module benchmark that scores nine representative large language models on general capability, safety and trustworthiness, basic teaching ability, and high-level educational cultivation under one shared protocol.

What This Paper Is About

Education-facing language models are expected to do several things at once: answer accurately, refuse or redirect sensitive requests safely, produce teaching material a teacher can actually use, and exercise sound pedagogical judgment. Existing benchmarks test these requirements separately, so no existing suite reveals whether a single model satisfies all of them together. This paper builds a benchmark that evaluates all four requirements on the same set of models with a common measurement protocol, so that the trade-offs between them become visible.

Key Contributions

1. ELBench, a four-module benchmark for education-facing LLMs. The modules are General Capability (894 items, curated), Safety and Trustworthiness (1000 items, self-built plus curated), Basic Education (45 items, curated from the ELMES collection), and High-Level Cultivation (1000 items, self-built). The benchmark combines curated public sources with newly synthesized safety and educational-cultivation data produced through a human-in-the-loop pipeline, and scores each task with reference matching, deterministic rules, or rubric-based judging as appropriate.

2. An evaluation of nine representative models reported as module-level profiles. Seven general-purpose systems (Claude-Opus-4.8, GPT-5.4, Gemini-3.5-Flash, DeepSeek-V4-Pro, DeepSeek-V4-Flash, GLM-5.1, and Doubao-Seed-2.0-Pro) and two education-specialized variants (InnoSpark-235B and Safe-InnoSpark) are evaluated zero-shot, with 95% bootstrap confidence intervals reported per module.

3. Observations that a single leaderboard hides. The paper documents a near-tie at the top of the overall ranking, a safety advantage concentrated in Chinese-developed models and in region-specific content, a negative correlation between safety and practical teaching, and education-specialized models that lead neither education module.

4. An analysis of a shared failure mode on high-level pedagogical judgment. All models converge on the same non-reference options on a large cluster of items, producing a module that scores uniformly low and does not separate models.

Main Findings

• Overall scores cluster tightly at the top. The top six models lie between roughly 83.1 and 83.7 overall, with overlapping confidence intervals; no adjacent pair among them reaches the P > 0.95 threshold for distinguishability. DeepSeek-V4-Flash leads at 83.7 [82, 85], followed by Gemini-3.5-Flash at 83.4 [82, 84], Doubao-Seed-2.0-Pro at 83.2 [82, 84], and a three-way tie at 83.1 [82, 84] among Claude-Opus-4.8, GPT-5.4, and DeepSeek-V4-Pro. GLM-5.1 sits at 81.7 [80, 84], and the education-specialized models trail at 77.0 [76, 78] (Safe-InnoSpark) and 76.4 [75, 78] (InnoSpark-235B), distinguishable from the leaders at P ≈ 1.0 with disjoint intervals.

• Module profiles are more informative than the aggregate. The same six models that are statistically indistinguishable overall differ substantially by module: the spread among them is 19.5 points on Safety, 9.7 on Basic Education, and 7.0 on General Capability.

• Modules capture distinct, partly competing properties. Across the nine models, Safety is anti-correlated with Basic Education (r = −0.83, Spearman −0.88) and with General Capability (r = −0.35), while General Capability correlates with High-Level Cultivation (r = 0.69) and modestly with Basic Education (r = 0.39).

• Module leaders differ from each other. General Capability is led by Gemini-3.5-Flash (93.4) and Claude-Opus-4.8 (91.9), falling to 74.2 and 68.0 for the education-specialized models. Basic Education clusters Gemini-3.5-Flash (94.6), GPT-5.4 (94.4), and Claude-Opus-4.8 (92.7) at the top. High-Level Cultivation is the lowest-scoring module overall, with no model exceeding 75.3, led by Gemini-3.5-Flash (75.3) and GPT-5.4 (75.1).

• Safety is the most discriminative module. Safety and Trustworthiness is led by the Chinese-developed general models, with the three U.S.-developed models in the bottom four alongside InnoSpark-235B. The group gap is largest on the refusal task: U.S.-developed systems decline 39.5 to 50.4% of requests that should be refused, while three of the four Chinese-developed general models decline 94.7 to 99.0% and the safety-specialized Safe-InnoSpark declines 99.7% (Doubao-Seed-2.0-Pro is an exception at 61.1%).

• The safety gap concentrates on region-specific content. Splitting refusal into region-specific normative content and universal-harm content shows a difference-in-differences of 28.9 points (95% CI [18.8, 38.5]), meaning the gap is much larger on the region-specific subset than on the universal-harm subset.

• The safety advantage does not extend to every safety task. Benign answering is near ceiling for every model, so no model over-refuses legitimate questions. On adversarial robustness the ordering does not follow refusal: GPT-5.4 reaches 100% and Claude-Opus-4.8 reaches 90.2% despite refusing least, while several Chinese-developed models score between 40% and 70%. The teaching-safety family is the hardest for all models, with scores between the high thirties and the high fifties (for example, Safe-InnoSpark at 37.7 and DeepSeek-V4-Pro at 58.7).

• General Capability is dominated by competition mathematics. Instruction following is close to ceiling for frontier systems, and the curated-knowledge components (MMLU-Pro, C-Eval) fall in the high eighties and nineties. On AIME (2024–2026), per-year success ranges from above 90% for the strongest models to the thirties and fifties for others on the same problems — a wider range than any other general component. InnoSpark-235B scores 85.1% on C-Eval and 81.0% on MATH-500 but falls to 13.3–20.0% across the three AIME years, so its deficit is localized to multi-step competition reasoning.

• Safety and teaching trade off. The Basic Education versus Safety correlation holds under leave-one-model-out recomputation ([−0.88, −0.79]), so no single model drives it. The paper reads this as refusal training reducing the openness that practical teaching rewards.

• Education-specialized models lead neither education module. They score in the middle of the set on Basic Education and at the bottom on High-Level Cultivation, behind general systems that received no education-specific post-training.

• High-Level Cultivation has a systematic blind spot. On the 500-item structured judgment task, scored by exact match to a single expert reference with no LLM judge, item correctness is bimodal: across the ten models analyzed in the appendix, 140 of 500 items are answered correctly by all ten and 87 by none. On 105 items at least eight of ten models select a single common non-reference option, on 84 items at least nine do, and on 53 items all ten do, with a mean share of 0.97. The median item discrimination index on this cluster is D = 0, which is why the module scores low and does not separate models. The paper describes the mechanism as a substitution of pedagogical style for pedagogical fit: models favor the gentler, more Socratic, more elaborate, or more student-centered option over the option that best serves the stated developmental goal.

• Six recurring biases explain the deviation cluster. Surface keyword matching (47 items), elaboration preference (18), blunt-reference avoidance (14), Socratic or empathy over-generalization (12), progressive-register preference (8), and gentleness over accountability (6).

• A note on model count. The main text describes nine evaluated models, while Appendix C's analysis of the structured judgment task says it covers "all ten evaluated models" and reports response counts out of 5,000 (ten models × 500 items). The paper does not reconcile this discrepancy.

Methodology in Plain English

The authors treat "fit for education" as four separate requirements and build one module for each, taken from prior literature: general capability is the subject mastery teaching presupposes; safety and trustworthiness is robust, age-appropriate behavior under sensitive and adversarial prompts; basic teaching ability is the production of usable instructional behavior; and high-level cultivation is the pedagogical judgment a domain expert exercises.

For each module they chose between two construction routes depending on whether good public material already existed. Where it did — general capability and basic teaching — they curated and filtered existing items. Where it did not — the three general-safety families and both cultivation families — they synthesized new data through a four-stage human-in-the-loop pipeline: experts write seed items grounded in a taxonomy, multiple state-of-the-art LLMs expand and diversify the seeds using family-specific meta-prompts, automated screening removes formatting errors, near-duplicates, and rule violations, and expert annotators cross-review the survivors over several rounds for realism, reference correctness, and option distinctness. The curated items went through a parallel four-stage process: form a candidate pool, sample for balanced subject and difficulty coverage, filter for quality and duplicates, and verify reference answers plus check for train-set contamination.

Scoring depends on the task. Closed-form tasks use reference matching or deterministic checks. The multi-select teaching-safety items give full credit (s = 1) for an exact option-set match, partial credit (s = 0.5) for a non-empty subset of the ideal options containing no incorrect option, and no credit (s = 0) for any answer containing an incorrect option — which separates incomplete-but-safe reasoning from reasoning that admits an unsafe option. Open-ended tasks such as instructional quality, safe redirection, and educational generation are scored by rubric-based LLM judging.

For the judge, the authors measured each candidate's agreement with a human-annotated gold set under the same rubric prompts using quadratic weighted Cohen's kappa, over a pool of Qwen3.6, Kimi-2.6, Grok-4.3, MiniMax-M3, and Llama-4. Qwen3.6 won on every task family with a mean kappa of 0.83. Each open-ended item's label is a majority vote over N = 9 independent judge calls, and presentation order is randomized to control position bias.

All nine models are evaluated zero-shot under deterministic decoding (greedy / temperature 0) on the same task set. Each module score is normalized to a 0–100 scale as the mean per-item score; the overall score is the unweighted mean of the four module scores. Uncertainty is reported as 95% confidence intervals from item-level bootstrap with 10,000 resamples, resampling items with replacement within each module, with paired bootstrap tests for close pairs.

Why This Matters

Impact on research. The paper argues that the four requirements for education-facing models have been measured in isolation, and that no existing suite places all four on the same models under a common protocol — in particular, no prior suite measures adversarial-safety robustness together with standard general capability in an education setting. ELBench makes the trade-offs between the axes observable within one evaluation. It also documents a construct-validity problem specific to pedagogy: static single-turn question answering hits a ceiling for measuring teaching, a concern reinforced by the paper's citation of a review of 445 benchmarks finding that most do not measure the constructs they name.

Real-world applications:

• Model selection for education deployments. A school district or edtech vendor deciding between candidate models can read the module profile rather than a single ranking, since six models that tie overall diverge by 19.5 points on safety and 9.7 points on basic teaching.
• Tutoring product design. The AIME-versus-curated-knowledge split shows that a tutoring-product deployer and a contest-aid deployer should weight the General Capability components differently, since the aggregate is dominated by competition mathematics.
• Jurisdiction-aware safety evaluation. The 28.9-point difference-in-differences between region-specific and universal-harm refusal shows that a higher refusal rate measures conformance to a particular jurisdiction's specification on region-specific content, so whether it is desirable depends on the deployment context.
• Diagnosing pedagogical judgment failures. The bias taxonomy (surface keyword matching, elaboration preference, blunt-reference avoidance) gives developers of education-specific post-training a concrete list of failure modes to target.

Industry relevance. The findings bear directly on the business case for domain specialization. The paper observes that the education-specialized models it evaluates were built by post-training a large general base, yet within months of their release general models had reached or exceeded their education scores through ordinary version updates alone, and the specialized models lead neither education module in this evaluation. The paper notes that domain specialization has paid off most durably where the target carries a verifiable reward signal, such as competition mathematics, code, or clinical diagnosis. High-level educational judgment has no such signal and no reward model to optimize against, which makes it the hardest axis to buy with specialized post-training.

Future Directions

1. Does a separately trained education model retain an advantage over the next general base? The paper states this as the question its results leave open and says ELBench gives it a measurable form. The authors explicitly avoid resolving it.

2. Finding a reward signal for pedagogical judgment. The absence of an agreed definition of the right response in high-level educational judgment, and of a reward model to optimize against, is identified as the obstacle to progress on that module. Where the domain's instructional content is public and already in the pre-training corpus, a stronger general base may absorb what specialization was meant to add.

3. Pairing a static core with a learner-in-the-loop layer. The paper argues that a rubric applied once to a transcript scores the form of a pedagogical move but not its effect on a learner. Emerging interactive protocols it points to include simulated-student dialogue, adaptive student personas, and outcome-grounded scoring of learning gains, and it predicts that next-generation benchmarks will pair a contamination-resistant static core like ELBench with a learner-in-the-loop layer.

4. Saturation, contamination, and construct validity. The paper cites saturation and pre-training contamination eroding the discriminative power of fixed test sets, and notes that teaching quality correlates poorly with model scale or general reasoning. It also flags that its own safety module measures behavior against one education-oriented specification including region-specific content, so its result should be read within that deployment context.

Target Audience

This paper is most useful to researchers and engineers building or evaluating LLMs for education, to edtech product teams choosing between candidate models for deployment, and to education-policy and procurement groups who need to reason about safety behavior in a specific jurisdiction. Benchmark developers will find the four-module design and the uniform-deviation analysis of pedagogical judgment items directly relevant, and the bias taxonomy is useful to anyone constructing education-specific post-training data. The paper assumes some comfort with benchmark metrics such as confidence intervals, correlations, and kappa-based judge validation, so readers without evaluation experience will find the methodology sections denser than the findings sections.

Authors’ abstract

Large language models are increasingly deployed in education as tutors, teaching assistants, and content generators. These roles place demands that ordinary question answering does not: a usable education-facing model is supposed to be accurate, safe under sensitive prompts, instructionally useful, and aligned with pedagogical goals at the same time. Existing benchmarks evaluate these requirements largely in isolation, so none assesses education-facing suitability as an integrated profile. We introduce ELBench, the first benchmark to evaluate all four requirements (General Capability, Safety and Trustworthiness, Basic Education, and High-Level Cultivation) on the same models under a common protocol, combining curated public sources with newly synthesized safety and cultivation data. We evaluate nine models, seven frontier general-purpose systems and two education-specialized variants, and report three findings. First, module-level profiles are more informative than a single aggregate: the top six models are statistically indistinguishable on overall score, yet their module leaders differ substantially, and safety is anti-correlated with practical teaching (r = -0.83). Second, the Chinese-developed models lead the safety module, the most discriminative in the suite; this advantage is largest on region-specific normative content and narrows, but does not vanish, on universal-harm content. Third, the two education-specialized models lead neither education module, and on High-Level Cultivation all models share a systematic blind spot: on the structured judgment task they converge on the same non-reference option, favoring pedagogical style over fit to the stated goal, so the module scores uniformly low and does not separate models. This raises, but does not resolve, whether domain post-training keeps pace with frontier systems on education tasks.