EduClaw-Bench: A Long-Horizon Benchmark for Pedagogical LLM Agents with Simulated Learners

AI summary

Overview

Research area: AI evaluation and AI safety/ethics for education — specifically benchmarks for autonomous LLM tutor agents operating over a learning management system (LMS).

Technical level: Intermediate. The paper combines knowledge tracing, simulated learners, LLM-as-judge rubrics, and agent harness engineering, but explains each component in accessible terms.

Scope: The paper introduces EduClaw-Bench, a 30-day virtual benchmark that pairs agent tutors with a knowledge-tracing-grounded simulated learner and scores them on three outcome axes plus two curriculum-design axes across 55 scenarios and three base-model tiers.

What This Paper Is About

Most educational LLM applications are point solutions to single tasks, and existing tutor benchmarks score individual turns or sessions against fixed learner utterances rather than measuring the learning a tutor actually causes. Because learning unfolds over days and weeks, evaluating a tutor requires a learner that persists, updates its knowledge in response to tutoring, and can be probed for gain. EduClaw-Bench addresses this by placing an agent tutor in a continuous 30-day relationship with a simulated learner whose per-knowledge-concept mastery comes from a knowledge tracing (KT) model trained on real-student data, and by scoring the tutor on near-independent axes.

Key Contributions

1. The benchmark itself: EduClaw-Bench places an agent tutor in a 30-day, LMS-based relationship with a KT-grounded simulated learner, scoring learning gain, responsiveness, helpfulness, and two curriculum-design rubrics across 55 persona-and-schedule scenarios.
2. A finding about attribution: Tutoring quality belongs to the base model and the agent harness together rather than either alone, so rankings reorder across tiers and single-tier leaderboards mislead.
3. A finding about horizon: Almost no model-and-harness combination sustains tutoring over the full horizon — learning plateaus within days — so long-horizon competence needs a purpose-matched combination.
4. A two-sided reliability check: The simulated learner's calibration (ECE = 0.049) and a live-classroom field study with about 500 K-12 students support trusting the measurements, with stated limits on each.

Main Findings

• No adapter leads learning gain on more than one tier: openclaw leads on Solar-pro3 (+0.36%), zeroclaw on Codex-gpt5.5 (−0.08%), and metaclaw on Qwen3-4B-Thinking (+0.64%). Base model and harness interact rather than contributing separably, so a single-tier leaderboard mis-ranks the same systems.
• Responsiveness runs inversely to Helpfulness on the frontier tiers: responsiveness spans 25–100%. The always-answer adapters (picoclaw, deeptutor, ironclaw) score lowest on Help (3.8–4.6) while the withholding adapters (openclaw, zeroclaw) score highest (5.6–6.1). Direct leakage stays negligible at 0.05% hand-over.
• Every agent plateaus within 5–10 days far below steady learning: on the Codex tier, per-day student accuracy flattens well below the dashed ideal-learning reference (openclaw moves from 0.28 to 0.29).
• Curriculum structure rarely forms: ironclaw and metaclaw lead on the frontier tiers (Gagné up to 1.93), but on the Small tier only metaclaw keeps measurable structure (Gagné 1.74) and the rest fall to 1.24–1.47, near the "event absent" floor. The best scores of 1.93 and 1.90 are against a 5.0 ceiling. Gagné and Rosenshine agree closely (Spearman ρ > 0.97).
• Failures are structural, not leakage: across runs, answer disclosure is essentially absent (≈ 0%), neglect affects 11.7%, but no-curriculum (Gagné below 1.5) affects 48.5% and no-learning-gain (Δ Solve Rate ≤ 0) affects 53.3%. metaclaw is the only adapter to mostly escape the no-curriculum mode (10% of runs), while zeroclaw hits that mode on 82% of runs.
• The simulated learner tracks real students: binning all 176,187 (run, day) pairs by predicted mastery against observed accuracy gives an Expected Calibration Error of 0.049 and a Brier score of 0.033 over 1.19M item attempts.
• The primary axes are largely independent: on openclaw over the n = 30 scenarios containing a help-request, pairwise Spearman correlations are ≈ 0.01 (I, II), ≈ −0.12 (I, III), and ≈ −0.13 (II, III), so a scalar composite would discard signal.
• RFT collapses one adapter's pedagogy while leaving responsiveness intact: on the Small tier with LoRA (rank-16, 5 epochs), metaclaw-rft holds Help nearly flat (4.73 to 4.76) while openclaw-rl-rft falls from 3.19 to 2.30 (per-epoch 3.18 to 2.50 to 1.85 by epoch 3, with no recovery). Because neither loses responsiveness, the collapse is invisible to leak- or refusal-based checks.
• Human validation is mixed but positive: panel–expert Spearman agreement reaches the expert ceiling on answer-holding (0.82 against 0.85 expert–expert) and tracks the other axes as a more modest proxy: Helpfulness 0.33 (expert ceiling 0.91), Gagné 0.43 (ceiling 0.64), Rosenshine 0.49 (ceiling 0.90), with permutation p < 0.05 on every judged axis.
• The rubric transfers to real classrooms: about 500 K-12 students used a lightweight LMS four times over two weeks, tutored by a stage-one regex claw on solar-mini. Scoring 150 of these sessions gives a field Helpfulness of 6.02 (sd 1.21), inside the simulator's range of 6.19 (sd 0.45), with no distinguishable difference (Mann-Whitney p = 0.68), and answer leakage matches at 0.05% in both settings.
• Reliability degrades sharply across students: pass 1 is near 0.47 on every tier, collapsing to pass 4 near 0.11 — a tutor that helps the average student less than half the time almost never helps all four.
• The difficulty-adaptation affordance goes unused: the adjust_level endpoint has a frequency of exactly 0 for all ten adapters. Provide_content, by contrast, spans two orders of magnitude, from metaclaw (138.7 per run) to ironclaw (98.3 per run).

Methodology in Plain English

The researchers built a virtual classroom that runs for 30 virtual days. Each of the 55 scenarios is one of 11 learner personalities (including "confused productive," "silent struggler," and "frustrated spiral," plus one adversarial student used for safety probes) combined with one of 5 study schedules (after-school, homework-only, school-integrated, weekend-warrior, self-directed), and each scenario is defined in a YAML file that fixes daily blocks, LMS event triggers, and probe schedules.

The learner has two coupled parts. First, a knowledge tracing model — an Attentive Knowledge Tracing (AKT) model trained on the XES3G5M elementary-mathematics interaction dataset (KC-level split, 5,027 sequences truncated to 200 interactions each, 7,652 questions, 865 knowledge concepts, 5-fold learner-disjoint split, test AUC 0.80). At inference it maintains a belief over K = 831 knowledge concepts that is updated from every graded answer. Second, an LLM that role-plays the student and produces free-response attempts, conditioned on that belief and on the tutor's context. Each scenario draws from 2,169 persona seeds pooled from XES3G5M's training students, each carrying a 200-interaction warm-up history used to precompute an initial mastery belief (one-time, roughly 30 minutes on GPU for the full 2,169 × 831 matrix). A pre-experiment held items and tutor context fixed and found the three ability levels produced significantly different pre-tutor probe accuracies (mean 0.19 / 0.34 / 0.53 on a 200-item set), so ability level is a real experimental knob.

The tutor sits behind a fixed interface: five LMS read endpoints, three LMS write endpoints, and a symmetric messaging channel. Only writes and messages count as actions, yielding exactly five observable action kinds: send_message, stay_silent, assign_items, adjust_level, and provide_content. Queries are prerequisites, not interventions. All state is written to a per-run trace log so judge scoring can be re-run offline.

Scoring uses five axes. Axis I is the mean per-day learning gain (Δ Solve Rate), inherited from prior pedagogical-RL work. Axis II is responsiveness, the fraction of student help-requests answered, detected by deterministic string match because student utterances come from a fixed template set — so this axis is exact, with no judge variance. Axis III is per-day helpfulness, the mean of 29 LearnLM rubric item scores on a 1–10 scale. Axes IVa and IVb score each day (from day 2) on Gagné's Nine Events of Instruction and Rosenshine's Ten Principles of Instruction, both on a 1–5 scale where 1 means "event absent."

Every judged axis uses a cross-family panel of three LLM judges (gemini-3-flash, gpt-4.1-mini, kimi-k2.5), run with reasoning disabled at temperature 0.3, taking the panel mean for Helpfulness and the curriculum rubrics and the panel majority for answer-holding. Three families guard against a family grading its own outputs when it appears as both student and judge.

The evaluation ran 10 agent adapters (one full agent framework, deeptutor; a metaclaw skill-proxy; and eight Claw-family gateway adapters with openclaw as the reference baseline) across three interchangeable base-model tiers: Solar-pro3 (Frontier), Codex-gpt5.5 (Frontier), and Qwen3-4B-Thinking (Small). A tier names the base language model an adapter runs on, not an agent system. Each tier used four LLM students of mixed provenance (solar-mini, llama-3.1-8b-instruct, qwen-2.5-7b-instruct, gemma-3-4b-it) to reduce single-family response bias. Runs used seed 42 with the student ability level fixed at "average." The nominal budget was 4 students × 13 adapters × 55 scenarios per tier; 8,637 completed runs were observed across the three tiers, of which 6,626 belong to the ten reported adapters, and the Small tier's LoRA trajectory study added 550 runs for 9,187 in total.

Why This Matters

Impact on research. EduClaw-Bench is the only benchmark in its own comparison table that pairs an agent tutor with a KT-grounded simulated learner and scores direct learning outcomes over a 30-day horizon across multiple base-model tiers on five axes. It shows that measuring learning requires a learner that actually learns, and that single-tier, single-session evaluation cannot surface the model-plus-harness interaction or the horizon plateau. It also documents that a low-weight helpfulness reward can collapse one adapter family's pedagogy mid-training while remaining invisible to leak- or refusal-based checks.

Real-world applications:

• Pre-deployment auditing for K-12 tutoring products: the authors present the benchmark as a gate a vendor can run before shipping a pedagogical LLM tutor to K-12 learners.
• Procurement and model-harness selection: because rankings reorder across tiers, buyers must evaluate the base model and harness pairing rather than reuse a single-tier leaderboard.
• Diagnosing structural failure modes: the four pedagogical failure modes (no curriculum, no learning gain, neglect, answer disclosure) give product teams specific targets rather than a single quality number.
• Validating evaluation instruments across settings: the field test shows the Helpfulness rubric returns comparable scores on real K-12 transcripts, supporting its use beyond synthetic dialogue.

Industry relevance. The paper releases all code, scenarios, per-run scores, and LoRA checkpoints, and explicitly recommends the benchmark as a pre-deployment gate rather than a substitute for supervised in-classroom evaluation. The unused adjust_level endpoint across all ten adapters suggests that difficulty adaptation remains an open engineering opportunity for harness developers.

Future Directions

• Same-system field deployment: the field study validated the instrument using a different tutor system (a stage-one regex claw on solar-mini) covering one subject, so it does not validate any specific adapter's rank. The authors leave a same-system field deployment to future work.
• Multi-seed confidence intervals: every run used seed 42, and the authors state that multi-seed confidence intervals are left to future work.
• Closing the model-plus-harness gap: since almost no combination sustains tutoring over the full horizon and learning plateaus within days, an open question is what purpose-matched combinations would convert the full 30 days into sustained gain.
• Exploiting unused affordances: the adjust_level endpoint was used at a frequency of exactly 0 by all ten adapters, raising the question of whether difficulty adaptation could improve outcomes on the other axes.
• Reliability over repeated runs: with pass 1 near 0.47 collapsing to pass 4 near 0.11, improving consistency across students is an unresolved target.

Target Audience

This paper benefits (1) researchers building or evaluating pedagogical LLM agents and long-horizon agent benchmarks, (2) educational technology developers and vendors who need a pre-deployment audit for K-12 tutoring products, (3) learning scientists and instructional designers interested in how Gagné's and Rosenshine's frameworks can be operationalized as automatic rubrics, and (4) AI safety and evaluation researchers interested in calibration of simulated users, cross-family LLM judge panels, and failure modes that leak- or refusal-based checks miss.

Authors’ abstract

Large language models (LLMs) power educational applications from tutoring to essay scoring, but each is a point solution to a single task, and only recently have these point solutions been integrated into agents operating over a learning management system (LMS). Yet tutoring is long-horizon, since a learner improves over days and weeks rather than in a single turn, and no benchmark evaluates an agent tutor across a sustained relationship. We introduce EduClaw-Bench, a benchmark that places an agent tutor in a continuous 30-day relationship with a simulated learner grounded in knowledge tracing (KT), whose knowledge-concept mastery, from a KT model trained on real-student data, drives its answers and is probed for learning gain across 55 scenarios. Each agent is scored on three primary axes (learning gain, responsiveness, and helpfulness) and two curriculum-design axes (Gagné and Rosenshine), with helpfulness and the curriculum axes judged by a cross-family panel of three LLM judges. Evaluating 10 agent adapters over three base-model tiers yields two findings that single-tier, single-session evaluation cannot reach. First, tutoring quality belongs to the base model and the agent harness together rather than either alone. Second, almost no combination sustains good tutoring over the full horizon. A calibration check ($\text{ECE}=0.049$) and a live-classroom field study confirm that the simulated learner and its measurements track reality. Our work is a step toward trustworthy AI tutors for future education.