Findings of the First Teaching Monster Challenge: A Benchmark of Pedagogical Content Knowledge in AI Agents

AI summary

Overview

• Research area: AI in education, specifically the evaluation of AI agents that generate instructional videos for a specified learner.
• Technical level: Intermediate. The paper is a benchmark and competition report rather than a new model architecture, but it assumes familiarity with LLM judging, Elo ranking, and instructional-design vocabulary.
• Scope: The paper reports the findings of the first Teaching Monster Challenge, a benchmark that scores AI-generated instructional videos on four dimensions using a three-layer protocol of an LLM-judge, crowdsourced pairwise comparison, and expert ranking.

What This Paper Is About

AI systems can now solve problems and answer questions like subject experts, but it is untested whether they can take a subject and reshape it so that a particular learner can understand it, which education research calls Pedagogical Content Knowledge (PCK). The authors introduce the Teaching Monster Challenge, the first instructional video generation benchmark that treats the learner persona as an explicit evaluation criterion: each system receives a course requirement plus a learner persona and must return one complete instructional video with no human in the loop. The paper reports the first edition's results, characterizing where current systems match expert pedagogical judgment and where they fall short. The version of the paper analyzed here ends mid-sentence in Section 8.2, so its appendices and any remaining discussion are not covered.

Key Contributions

1. A new task definition and benchmark. The Teaching Monster Challenge defines teaching as a single-shot generation problem: given a course_requirement and a free-text learner_persona, a system must produce one complete instructional video end to end without human involvement. Items span four secondary-level STEM subjects (Physics, Biology, Computer Science, and Mathematics) anchored to the Advanced Placement curriculum, with a matched-pair subset that holds the course requirement fixed and varies only the persona.
2. A four-dimension PCK rubric. Submissions are scored on Content Accuracy, Pedagogical Logic, Learner Adaptability, and Engagement and Multimodal Presentation, with the first three mapped onto Shulman's three kinds of teacher knowledge.
3. A three-layer evaluation protocol. Layer 1 uses a three-agent LLM-judge running on gemini-3-flash against a 22-item rubric; Layer 2 uses crowdsourced blind pairwise comparison aggregated into Elo ratings; Layer 3 uses a panel of 10 secondary-school teachers, school principals, and university professors to produce the final ranking.
4. Released assets and an audit of automatic judging. The paper releases the benchmark, rubric, and human judgments as a testbed, and documents where the LLM-judge diverges from human preference.

Main Findings

• Content is stronger than delivery. Mean Layer 1 scores across all submissions were 4.31 for Content Accuracy and 4.19 for Pedagogical Logic, versus 3.77 for Learner Adaptability and 3.86 for Engagement. Adaptability was lowest overall, and the pattern held for most individual systems.
• Challenge outcome. tsunumon won the challenge, followed by OmegaZeroRe and ScaffoldAI. The Layer 3 experts agreed at Kendall's W = 0.41. The crowd ranked tsunumon first but could not order OmegaZeroRe and ScaffoldAI, whose Elo intervals overlapped.
• The LLM-judge screens well but ranks poorly. The judge's ranking of the ten shortlisted systems had almost no agreement with the Layer 2 crowd ranking (Spearman ρ = -0.17). tsunumon, the eventual winner, ranked only fourth under the judge, while the judge's top two, Team 67 and Kuro, finished eighth and tenth with the crowd. The judge's scores for the ten are clustered near the ceiling of 5, leaving little room to separate them.
• The low-performing tail is real. Figure 3 shows a long tail of weak or off-topic submissions scoring well below the rest; removing it let the costly human evaluation focus on the ten shortlisted systems rather than all 77 teams.
• The baselines beat nearly everything. Three organizer baselines were scored with the same Layer 1 judge: a cascaded pipeline (4.52 Accuracy, 4.70 Logic, 3.64 Adaptability, 4.12 Engagement), a commercial black-box generator (Google NotebookLM Video Overview, accessed May 2026; 4.53, 4.48, 4.00, 4.26), and a static retrieval framework (2.12, 2.47, 3.53, 4.01). Under Layer 2, all three placed between first and second. Only the winner tsunumon beat the baselines, so systems below it had not clearly improved on a mature commercial product or a well-chosen human-made video.
• The static retrieval baseline's low scores measure matching, not teaching. Its deduction records show most penalties come from videos missing or only partly covering the requested topic, not from factual errors.
• Deduction patterns. Of 6,699 deduction flags across 1,612 preliminary-round videos, visual delivery drew the largest share (39%), led by ineffective visual representation, which draws a deduction in a third of videos. Learner adaptation was 27%, driven mostly by missing scaffolding, jargon overload, and prerequisite gaps. Content problems were rarer (16%) and mostly concerned coverage and depth rather than correctness, with narration issues adding 8% and explanation errors 10%. About one video in six still contained a critical factual error.
• Systems do adapt, and people can see it. In a persona-identification study, raters chose the intended learner from three candidates at 30.8% for the persona-independent baseline (near the 33.3% chance rate), 51.7% for the experimental group of shortlisted systems, and 61.9% for the human-video group. The gap between baseline and experimental group held under a mixed-effects logistic regression with random intercepts for video and rater. The human-video advantage over the experimental group was not statistically significant.
• Judge and crowd emphasize different things. The judge records criteria by prevalence (how often a criterion is registered across 320 videos), while the crowd cites reasons by salience (the share of 145 written rationales mentioning a criterion). Some judge criteria, such as pure-calculation bias, are recorded on 99.7% of videos and almost always favorably, so they separate nothing.
• A qualitative case study of a finalist video on Mendelian genetics found accurate but imprecise wording (the phrase "4 genotype boxes" conflates Punnett-square cells with genotype categories, since an Aa × Aa cross has four equally likely gamete-pair outcomes but only three genotype categories), unexplained visual choices, an unexplained shift from ABO blood type to a generic Aa × Aa cross, introduced-then-unused terminology, and no cursor or pointer to guide attention.
• Participant feedback. Of the 15 teams responding to a questionnaire, feedback indicated that the end-to-end challenge design was highly valued, while primary engineering frictions stemmed from multimodal synchronization under strict latency constraints and ambiguous LLM-judge feedback.
• System design convergence. A post-challenge questionnaire showed nearly all submissions instantiate a six-stage cascade: persona parsing and routing, lesson planning, slide and script composition, visual rendering, narration synthesis, and assembly.

Methodology in Plain English

The organizers designed a benchmark where every system sees the same two inputs: a course requirement stating the learning objective and key concepts, and a free-text learner persona describing the learner's background, prior knowledge, and gaps. Items cover four secondary-level STEM subjects and are anchored to the Advanced Placement curriculum. A subset forms matched pairs, where the course requirement stays fixed and only the persona changes, so any difference between a system's two videos is attributable to the manipulated persona rather than topic difficulty or baseline production quality.

Measuring actual learning gains from each video is infeasible at benchmark scale, so the organizers evaluate teaching quality as a proxy for learning, following established practice in which observer ratings of teaching quality predict achievement gains and design-quality ratings of instructional videos relate to how much learners take from them.

Evaluation runs in three narrowing layers. Layer 1 screens every submission with an automated LLM-judge using a three-agent architecture adapted from EduPanel, running on gemini-3-flash against a rubric of 22 items, producing a 1-to-5 score on each of four dimensions; systems are ranked by the mean of those four scores. Layer 2 takes the ten highest-ranked systems into a blind Arena-style pairwise comparison done by raters recruited through Prolific, who are shown the persona and requirements and asked which of two videos better helps the specified learner; outcomes are aggregated into Elo ratings with 90% bootstrap confidence intervals. Layer 3 takes the top three systems, whose videos for expert-designed items are ranked directly by an expert panel of 10 secondary-school teachers, school principals, and university professors, with each subject covered by three panel members (two in the subject matter, one in pedagogy). Each system's average rank across the experts determined the final standings.

Competition logistics ran in three phases. A Warm-up calibration phase had two rounds starting March 1, 2026 (LLM-generated requests) and April 14, 2026 (committee-authored items), drawing 46 teams and 1,696 videos. The Preliminary phase began May 1, 2026 with two rounds of 16 items each, drawing 77 teams and 1,612 videos. The Final phase began June 12, 2026, where the three finalists generated 48 videos for 16 expert-designed items. Submissions ran through an organizer-built platform with a 30-minute generation budget per API request; over-budget responses were treated as failures. Award-candidate teams had to submit a complete system image, such as a Docker image, for post hoc verification in an isolated environment.

For the persona-study component, raters watched one video and chose among three candidate personas matched in length and format and shown in random order, spanning three groups of videos over the same eight topics, two from each subject.

Why This Matters

Impact on research. Prior benchmarks measure problem-solving accuracy, text-dialogue tutor responses, or perceptual video quality, and none treats adaptation to a particular learner as an explicit criterion. This work fills that gap and, unusually, audits its own automated judge against human preference, showing that the judge separates a clear low-performing tail but fails as a final ranker. That finding means progress requires not only better teaching systems but also better automatic judges.

Real-world applications:

• Personalized instruction at scale, addressing the shortage of teachers that UNESCO estimates will reach 44 million more needed by 2030.
• Automated generation of supplementary instructional videos matched to individual students' grade level, prior knowledge, and attention span.
• Benchmarking and procurement of commercial AI teaching products, since the paper shows that a mature commercial generator placed between the first- and second-ranked competition systems.
• Evaluating teacher-training and instructional-design quality using the four-dimension rubric as a structured review instrument.

Industry relevance. The competition field was large (77 submitting teams in the Preliminary phase alone) and the design space has already converged on a recognizable six-stage pipeline, with recurring choices such as JSON persona conditioning, difficulty-based model routing, web-search grounding of lesson plans, Bloom's-taxonomy constraints enforced by closed-loop LLM review, HTML/CSS slide templates with reviewer–fixer loops, and forced alignment of narration to animation via word-level timestamps. The finding that nearly all non-winning systems failed to beat baselines is a direct signal to builders about the current performance bar.

Future Directions

1. Build better automatic judges. The paper states that progress requires better automatic judges as well as better teaching systems; the current judge's near-ceiling scores on the strongest systems leave it unable to rank them.
2. Improve delivery and adaptation rather than content. Visual delivery and learner adaptation dominate the deductions, and Adaptability was the lowest-scoring dimension, while content scores are already high.
3. Reduce remaining factual errors. About one video in six still contains a critical factual error despite high content scores, so explanation mistakes coexist with high aggregate accuracy.
4. Address the engineering frictions reported by participants. The two named frictions are multimodal synchronization under strict latency constraints and ambiguous LLM-judge feedback.
5. Extend toward interactive tutoring. The paper explicitly scopes out interactive tutoring, where a teacher responds to a learner's questions and errors, describing it as too open-ended to compare across systems.

Target Audience

Researchers in AI and education who study instructional content generation and evaluation; builders of educational AI agents and instructional video pipelines; developers of automated evaluation and LLM-as-judge systems who need evidence about where automatic judging breaks down; and education practitioners, curriculum designers, and teacher-training specialists interested in how well current AI systems adapt material to a specified learner.

Authors’ abstract

AI agents can now solve problems, answer like subject experts, and generate long-form multimodal content. However, whether they can adapt a lesson to fit a specified learner, which education calls Pedagogical Content Knowledge (PCK), has not been benchmarked. To measure it, we introduce the Teaching Monster Challenge, the first instructional video generation benchmark to treat the learner persona as an explicit evaluation criterion. Each system is given a topic and a learner persona and must generate a complete instructional video. Every video is screened by an LLM-judge, ranked by crowd pairwise voting, and finalized by an expert panel. The first edition shows that today's systems handle the content well but are far weaker at presenting it and adapting it to the learner. The same process exposes a limit of automatic judging. The LLM-judge separates a clear low-performing tail but ranks the strongest systems poorly. The strongest systems receive nearly identical scores from the judge, so its ranking of them does not match human preference. Progress therefore requires not only better teaching systems but also better automatic judges, and we release the benchmark, rubric, and human judgments as a testbed for both.