UCO: A Multi-Turn Interactive Reinforcement Learning Method for Adaptive Teaching with Large Language Models

AI summary

Overview

Research area: Educational applications of large language models, specifically multi-turn reinforcement learning for adaptive math tutoring.

Technical level: Advanced. The paper assumes familiarity with reinforcement learning (GRPO, POMDPs, reward shaping), policy optimization, and LLM post-training.

Scope: The paper introduces Unidirectional Cognitive Optimization (UCO), a multi-turn interactive reinforcement learning method that trains a teacher LLM with two cognition-oriented reward functions, and evaluates it against 11 baselines on BigMath and MathTutorBench.

What This Paper Is About

Most LLM tutoring systems are trained by supervised fine-tuning on teaching dialogues, which teaches surface-level patterns rather than dynamic adaptation. Existing reinforcement learning approaches reward the teacher only when the student outputs the correct answer, which cannot distinguish genuine understanding from guessing or echoing a revealed answer, and they cannot perceive how a student's cognitive state changes across turns. UCO's goal is to make a teacher model adapt its teaching strategy in real time by rewarding genuine cognitive progress and by keeping task difficulty inside each student's Zone of Proximal Development (ZPD).

Key Contributions

1. A unidirectional cognitive optimization method (UCO) that generates training trajectories online through iterative teacher–student interaction, adjusting teaching strategies as the student's cognitive state evolves, and updates the teacher policy with GRPO.

2. A cognition-oriented Progress Reward that approximates cognitive progress as entropy reduction, computed through two dimensions: a Potential Capability Score (log-probability the student assigns to oracle-generated correct responses, mapped through tanh into [-1, 1]) and a Semantic Quality Score (maximum cosine similarity between student output and candidate correct answers).

3. A ZPD-based Scaffold Reward that discretizes teacher behavior into five ordered scaffolding levels by cognitive load—Metacognitive, Strategic, Conceptual, Step-by-Step, and Example Demonstration—identifies the highest level at which the student succeeds, then targets one level below it as the ZPD, rewarding matches and penalizing proportional deviations.

4. A benchmark comparison against 11 baselines on BigMath and MathTutorBench, showing that a 7B UCO model surpasses all equivalent-scale models and performs comparably to leading closed-source models.

Main Findings

• BigMath balance: UCO reaches 30.2% Δ Solve Rate, 12.9% Leak Solution, and 4.6/4.5 Ped-RM micro/macro. The paper reports a 14.6 percentage point improvement in Leak Solution over the strongest teaching baseline, Qwen2.5-7B-RL, and a 28% improvement in Ped-RM scores.

• General models prioritize answers over teaching: GPT-4o achieves 33.1% Δ Solve Rate but has 35.2% Leak Solution and only 1.5/-0.3 Ped-RM. The Qwen2.5 series rises from 11.3% to 38.7% Δ Solve Rate as parameters grow from 7B to 72B, while Leak Solution rises from 29.3% to 61.0% and Ped-RM stays negative.

• Large models act as solvers, not teachers: DeepSeek-V3 reaches the highest Δ Solve Rate of 39.3% but shows 46.6% Leak Solution with -1.5/-0.8 Ped-RM scores.

• Education-specialized models trade off differently: SFT and MDPO have lower Leak Solution but Δ Solve Rates of 8.9% and 16.4%. SocraticLM and Qwen2.5-7B-RL balance better, with Qwen2.5-7B-RL at 29.1% Δ Solve, 15.1% Leak Solution, and 3.6/3.1 Ped-RM. LearnLM 2.0 Flash has the best Ped-RM (6.8/6.4) and lowest Leak Solution (0.9%) but only 4.3% Δ Solve Rate.

• MathTutorBench results: UCO scores 0.89 problem-solving accuracy, 0.24 Socratic questioning BLEU, 0.65 solution correctness F1, 0.45 mistake location micro F1, 0.08 mistake correction accuracy, and win rates of 0.68 (scaffolding), 0.74 (pedagogical IF), 0.72 (scaffolding hard), and 0.77 (pedagogical IF hard) against a human teacher.

• Scaffold reward matters most in ablation: Removing the scaffold reward on BigMath drops Δ Solve Rate by 6.7 points to 23.5%, raises Leak Solution by 7.3 points to 20.2%, and reduces Ped-RM to 2.2/2.3. Removing the progress reward drops Δ Solve Rate by 1.6 points to 28.6%, raises Leak Solution by 2.7 points to 15.6%, and reduces Ped-RM to 4.1/3.9.

• On MathTutorBench, removing the scaffold reward causes declines of 3.4% in Math Expertise, 9.0% in Student Understanding, and 10.6% in Pedagogy; removing the progress reward affects Student Understanding and Pedagogy more than Math Expertise.

• All reward configurations beat the backbone: the Qwen2.5-7B-Instruct backbone scores 11.3% Δ Solve Rate, 29.3% Leak Solution, and -0.2/-0.5 Ped-RM.

• Dense rewards beat sparse rewards: the paper states that the PedagogicalRL reward underperforms UCO across all dimensions under an identical training pipeline (same GRPO algorithm, 2,000 samples, 4 rollouts), despite its explicit LLM-based hard penalties.

Methodology in Plain English

The researchers treat tutoring as a partially observable Markov decision process where the teacher's action is a pedagogical strategy, the observation is the student's reply, and the hidden state is the student's cognitive state. A teacher model (Qwen2.5-7B-Instruct) and a fixed student model (Qwen2.5-14B-Instruct) interact over multiple turns to produce online rollouts. An oracle model (Gemini-2.5-Pro-Exp) generates candidate correct answers and one hint per scaffolding level, with K = 5 student responses sampled per turn.

After each turn, two rewards are computed. The Progress Reward measures whether the student is moving from confusion toward comprehension: it checks how much probability the student assigns to correct answers (potential capability, squashed through tanh with temperature α) and how semantically close the student's actual output is to correct answers (cosine similarity from an embedding model, with a bias term δ), then combines them with weight λ. The Scaffold Reward estimates which of five scaffolding levels the student can succeed at, sets the target one level lower to create "productive struggle," and gives a positive reward when the teacher's chosen level matches that target or a penalty proportional to the level distance (penalty coefficient c).

The two rewards are summed into a turn-level reward, discounted and aggregated into a rollout return, converted into an intra-group advantage, and used to update the teacher with GRPO with a KL penalty against a reference policy. Training used 2,000 randomly sampled BigMath instances over roughly 112 hours on 8 NVIDIA H200 GPUs with Python 3.11, CUDA 12.8, and PyTorch 2.7. Key hyperparameters include learning rate 5×10⁻⁶, batch size 16, G = 4 rollouts, maximum dialogue length T = 10, discount γ = 0.95, KL weight ξ = 0.01, normalization ε = 10⁻⁸, AdamW, λ = 0.5, α = 2.0, δ = 0.7, BGE embeddings, and c = 0.2.

Evaluation on BigMath follows the exact PedagogicalRL protocol using the same 500 held-out test questions, the same student model (Meta-Llama-3.1-8B-Instruct-FP8), and the same judge model (Gemma-3-27B-IT), replacing only the teacher. MathTutorBench evaluation follows the original paper's protocol across nine tasks in three domains.

Why This Matters

Impact on research: The paper reorients reward design for tutoring LLMs away from answer correctness toward cognitive state change, connecting pedagogical theory (Vygotsky's ZPD, Shannon entropy reduction) to computable RL signals. It provides an alternative to sparse, conversation-level rewards and to behavior-level rewards that measure teaching actions rather than student understanding.

Real-world applications:

• Adaptive math tutoring systems that avoid simply revealing answers while still improving student problem-solving.
• Teacher-training or teaching-assistant evaluation tools built on the Ped-RM and scaffolding-level framework.
• Automated generation of scaffolded hints at five calibrated levels for homework or practice platforms.
• Diagnostics of student confusion and mastery from dialogue, usable in intelligent tutoring systems.

Industry relevance: The results suggest that a 7B open model with cognition-oriented multi-turn RL can reach a three-way balance of learning gains, low answer leakage, and pedagogical quality that much larger general models do not achieve, which matters for the cost and deployment of educational AI products. The code is released at https://github.com/ECNU-RAIL/UCO-EMNLP2026.

Future Directions

• Extending the cognitive-state modeling and reward design beyond K-12 mathematics to domains such as programming education and scientific reasoning, which require domain-specific cognitive progress metrics, scaffolding hierarchies, and new benchmarks.

• Moving from text-only dialogue to multimodal teaching scenarios involving graphics, handwritten calculations, and vocal intonation, which the authors identify as a current limitation.

• Validating UCO in real classrooms, since effectiveness so far has been demonstrated mainly on offline datasets rather than with actual students.

• Building on the appendix analyses: testing alternative backbones, embedding models, and oracle choices, tuning hyperparameters such as λ, α, δ, c and the number of rollouts, evaluating on additional datasets such as MathDial, and running human evaluations. Note that the provided paper content is truncated mid-sentence in the appendix, so the full details and results of these additional studies are not reported here.

Target Audience

Researchers and practitioners in LLM post-training and reinforcement learning who work on educational applications; AI education engineers building tutoring or hint-generation systems; and learning-science or education researchers interested in operationalizing constructs such as the Zone of Proximal Development and cognitive progress as machine-computable reward signals.

Authors’ abstract

Large language models (LLMs) are shifting from answer providers to intelligent tutors in educational settings, yet current supervised fine-tuning methods only learn surface teaching patterns without dynamic adaptation capabilities. Recent reinforcement learning approaches address this limitation but face two critical challenges. First, they evaluate teaching effectiveness solely based on whether students produce correct outputs, unable to distinguish whether students genuinely understand or echo teacher-provided answers during interaction. Second, they cannot perceive students' evolving cognitive states in real time through interactive dialogue, thus failing to adapt teaching strategies to match students' cognitive levels dynamically. We propose the Unidirectional Cognitive Optimization (UCO) method to address these challenges. UCO uses a multi-turn interactive reinforcement learning paradigm where the innovation lies in two synergistic reward functions: the Progress Reward captures students' cognitive advancement, evaluating whether students truly transition from confusion to comprehension, while the Scaffold Reward dynamically identifies each student's Zone of Proximal Development (ZPD), encouraging teachers to maintain productive teaching within this zone. We evaluate UCO by comparing it against 11 baseline models on BigMath and MathTutorBench benchmarks. Experimental results demonstrate that our UCO model outperforms all models of equivalent scale and achieves performance comparable to advanced closed-source models. The code and data are available at https://github.com/Mind-Lab-ECNU/UCO.