EduDial: Constructing a Large-scale Multi-turn Teacher-Student Dialogue Corpus

AI summary

Overview

Research area: Natural Language Processing — conversational AI, large language model evaluation, and AI for education (intelligent tutoring systems).

Technical level: Intermediate. The paper is readable without deep technical background, but assumes some familiarity with supervised fine-tuning, preference optimization (DPO), and LLM benchmarking.

Scope: The paper introduces EduDial, a large-scale multi-turn teacher-student dialogue corpus grounded in educational theory, along with a fine-tuned model (EduDial-LLM 32B) and an 11-dimensional evaluation framework for measuring LLM teaching ability.

What This Paper Is About

Existing multi-turn dialogue benchmarks evaluate general conversational skill, but they do not test whether a model can actually teach — identifying what a student misunderstands, guiding reasoning step by step, and adapting to different learners. EduDial addresses this gap by building a teacher-student dialogue dataset that encodes real pedagogical structure, then using it to train and evaluate LLMs on teaching ability rather than on generic question answering.

Key Contributions

1. EduDial dataset. A corpus of 34,250 dialogue sessions covering 345 core K-12 mathematics knowledge points, built from a five-stage teaching process grounded in Bloom's taxonomy and ten questioning strategies (two per stage).

2. Differentiated instruction modeling. A four-role system — one teacher plus three simulated students (excellent, medium, struggling) — paired with role-specific teaching strategies, enabling evaluation of teaching in realistic multi-student classrooms rather than single-learner settings.

3. Two complementary data resources. The Multi-turn Teaching Instruction (MTI) set of 13,700 dialogues for supervised fine-tuning, and the Preference Dataset based on Teaching Strategies (PDTS) of 20,550 chosen/rejected response triplets for direct preference optimization.

4. EduDial-LLM and an 11-dimensional evaluation framework. A 32B model trained via SFT followed by DPO, plus a rubric spanning nine overall-quality dimensions adapted from the Classroom Assessment Scoring System (CLASS), and two content-quality dimensions (relevance and coverage).

Main Findings

• Most LLMs are weak at teaching, not just at conversation. Across 18 evaluated models, closed-source systems scored well on fluency and emotional support but poorly on pedagogical dimensions like thinking (THK) and interactivity (INT).

• Two recurring failure modes. The authors identify (1) lack of guided teaching — models such as DeepSeek-V3 answer directly instead of scaffolding reasoning — and (2) role consistency collapse, where models begin as teachers but degrade into generic QA systems as dialogue progresses.

• EduDial-LLM leads across all metrics. It achieves the highest average score in both machine (4.47) and human (4.36) evaluation, with balanced performance across all nine quality dimensions. Gains in thinking and insight trace to the SFT stage; gains in interactivity, adaptability, and feedback trace to the DPO stage.

• Both training stages matter, but asymmetrically. SFT alone substantially improves over the base model; DPO alone also helps but less so, primarily on thinking and adaptability. The combined two-stage approach outperforms either alone. Content quality (relevance and coverage) fluctuates little across configurations, suggesting knowledge delivery is easier than pedagogical interaction.

• Machine evaluation is a credible proxy for humans. Human-machine correlation averaged Pearson 0.90 and Spearman 0.91 across dimensions, with the highest agreement on objective dimensions (relevance, coverage, thinking) and lower agreement on context-heavy ones (interactivity, feedback, insight). Expert inter-rater reliability was Cohen's Kappa 0.76.

• Pedagogical skill transfers across datasets. In a zero-shot evaluation on the independent SocraTeach benchmark, EduDial-LLM beat GPT-4 (overall 0.57 vs. 0.50) and came close to SocraticLM (0.62), which had been trained in-domain. The untuned backbone scored 0.31, confirming the gains came from training data rather than the base model.

Methodology in Plain English

The researchers began by grounding the task in classroom practice rather than in generic dialogue generation. They surveyed over 100 primary and secondary schools, worked with experienced mathematics teachers, and selected 345 curriculum knowledge points across 173 chapters.

They then mapped the teaching process onto Bloom's taxonomy as five progressive stages: Introduction (remember), Concept Exploration (understand), Deep Understanding (analyze), Knowledge Application (apply/evaluate), and Reflection (create). Completing a stage signals that a student has absorbed that level, which is the natural moment to ask a question that pushes them to the next level. Each stage is assigned two specific questioning strategies — such as scenario-based, zone of proximal development, or metacognitive questioning — derived from analyzing MOOC videos and teacher discussions.

Three simulated student profiles (excellent, medium, struggling) were defined by cognitive capacity, motivation, engagement, and self-regulation, each with a matching teaching strategy: deep exploration for strong students, thinking guidance for mid-level students, and confidence building for struggling students. Dialogues were generated using o1 as a teacher-student simulator, then passed through expert-machine dual verification, where GPT-4o checked automatically and human experts reviewed against five teaching principles.

From verified dialogues, the MTI corpus was built for supervised fine-tuning (student turns as input, teacher turns as output, and vice versa for the student models). Separately, the PDTS corpus was built by deliberately generating "rejected" teacher responses that are plausible but ignore the prescribed strategy, alongside "chosen" responses that follow it, producing preference pairs for DPO. Training used QLoRA 4-bit quantization on two A100 GPUs, taking roughly 45 hours for SFT across four models and 8 hours for DPO.

Evaluation was conducted in both multi-student (one teacher, three students) and one-on-one settings, capped at 15 rounds, on knowledge points the model never saw during training.

Why This Matters

Impact on research. The paper argues that general-purpose multi-turn benchmarks like MT-Bench and AlpacaEval conflate conversational fluency with teaching competence. By operationalizing pedagogy into measurable dimensions, EduDial gives researchers a way to test whether a model can scaffold reasoning, correct errors, and adapt instruction — capabilities that domain-specific evaluation requires and that current benchmarks cannot detect.

Real-world applications:

• AI tutoring systems that decide when to ask a question rather than simply supplying the answer, which matters for developing independent reasoning.
• Classroom-scale deployment where one AI teacher serves students of mixed ability simultaneously, with distinct scaffolding for each.
• Curriculum-aligned content generation for lesson planning and exercise design, validated against coverage and relevance metrics.
• Teacher training and evaluation tools, using the 11-dimension rubric as an observation framework.

Industry relevance. The two-stage SFT-then-DPO recipe and the preference-pair construction method are directly transferable to any domain requiring strategy compliance rather than just plausible output. The finding that untuned backbones collapse into generic QA systems under sustained multi-turn interaction is a practical warning for anyone deploying LLMs in long-horizon advisory or instructional roles.

Future Directions

• Student-initiated inquiry. The current framework is teacher-led: the teacher questions, the student responds. Real learning also involves students raising questions and challenging ideas, which the authors flag as a gap.

• Multimodal teaching. The corpus is text-only, while human instruction relies on diagrams, step-by-step algebraic derivations, and vocal tone. Extending the framework to visual and auditory modalities remains open.

• Validation in real classrooms. Student agents respond according to predefined cognitive profiles. Testing whether the model's strategies hold up with actual students, with all their unpredictability, is necessary before deployment claims can be made.

• Robustness of automated evaluation. Machine ratings correlate highly with human ratings overall, but agreement drops on dimensions like interactivity and feedback that require contextual judgment — an open problem for scaling evaluation.

Target Audience

Researchers and engineers working on LLM-based educational agents, multi-turn dialogue benchmarks, or preference-based fine-tuning will find the most value here, particularly those interested in how domain-specific pedagogical theory can be encoded into training data. Educators and instructional designers interested in how AI systems might be evaluated against classroom observation frameworks like CLASS will also find the evaluation rubric useful. The paper is accessible to graduate students with basic knowledge of instruction tuning, though the data construction pipeline assumes some familiarity with LLM-based synthetic data generation and DPO.

Authors’ abstract

Recently, several multi-turn dialogue benchmarks have been proposed to evaluate the conversational abilities of large language models (LLMs). As LLMs are increasingly recognized as a key technology for advancing intelligent education, owing to their ability to deeply understand instructional contexts and provide personalized guidance, the construction of dedicated teacher-student dialogue benchmarks has become particularly important. To this end, we present EduDial, a comprehensive multi-turn teacher-student dialogue dataset. EduDial covers 345 core knowledge points and consists of 34,250 dialogue sessions generated through interactions between teacher and student agents. Its design is guided by Bloom's taxonomy of educational objectives and incorporates ten questioning strategies, including situational questioning, zone of proximal development (ZPD) questioning, and metacognitive questioning-thus better capturing authentic classroom interactions. Furthermore, we design differentiated teaching strategies for students at different cognitive levels, thereby providing more targeted teaching guidance. Building on EduDial, we further develop EduDial-LLM 32B via training and propose an 11-dimensional evaluation framework that systematically measures the teaching abilities of LLMs, encompassing both overall teaching quality and content quality. Experiments on 17 mainstream LLMs reveal that most models struggle in student-centered teaching scenarios, whereas our EduDial-LLM achieves significant gains, consistently outperforming all baselines across all metrics. The code is available at https://github.com/Mind-Lab-ECNU/EduDial/tree/main.