Sherpa: Teaching LLMs to Teach Adaptively

AI summary

Overview

Research area: Artificial intelligence in education, specifically reinforcement-learning post-training of large language models as adaptive tutors.

Technical level: Advanced. The paper assumes familiarity with multi-turn reinforcement learning, PPO-style policy optimization, and leave-one-out advantage estimation, though its central ideas are explained in accessible terms.

Scope: A multi-turn RL framework, Sherpa, that trains an LLM teacher by directly maximizing the measured learning gains of simulated student archetypes with differing pedagogical preferences.

What This Paper Is About

Existing ways of training LLM teachers rely on demonstrations, preference data, or hand-written rubrics describing what good teaching looks like, and none of these signals are grounded in how much an individual student actually learns. Because effective teaching strategies vary substantially across learners, the authors instead push the specification of instructional needs onto the student side, creating several student archetypes, and train the teacher purely on whether the student improves. The goal is an LLM tutor that infers a learner's needs from conversation and adapts its instruction accordingly.

Key Contributions

1. An outcome-based teacher training framework. Sherpa separates pedagogical specification from teacher optimization: diverse instructional needs are instantiated in the student, while the teacher is optimized only on the student's improvement at test time, rather than on predefined notions of good teaching.

2. A two-gate student archetype design. A guidance gate blocks any teacher instruction that reveals the ground-truth final answer, and an adaptive gate ensures that only guidance matching the student's assigned preference reaches the student backbone, exactly when the student needs it; rejected turns trigger a scripted complaint. This makes student preferences consequential and inferable, which prompting alone did not achieve.

3. Empirical gains across in-distribution and out-of-distribution students. Sherpa-trained teachers raise average student performance by 20.5 percentage points across all archetypes, including three preferences unseen during training, and generalize to MathTutorBench's pedagogy score (52.5% to 79.2%).

4. Human validation and design ablations. A study with 96 high school teachers shows the trained teacher is preferred in 79.6% of comparisons, and ablations show that adding predefined gate penalties degrades generalization, supporting the outcome-only training signal.

Main Findings

• Student performance improves for every archetype. With a Qwen3-8B teacher and Qwen3-1.7B student backbone on MATH problems, Sherpa DP (trained with diverse-preference students) reaches an overall test accuracy of 69.0%, compared with 48.6% for the untrained Qwen3-8B, 52.7% for PedagogicalRL, and 52.0% for Sherpa NP (trained with no-preference students). The student's overall unaided accuracy is reported as 36%.

• Diverse student archetypes matter. Sherpa DP outperforms Sherpa NP on every archetype, with in-distribution accuracy of 72.0% versus 54.8% and out-of-distribution accuracy of 65.1% versus 48.3%. Gains transfer unevenly: in the comparison between Sherpa NP and Sherpa DP, the Causal archetype moves from 55.1% to 77.1% and the Step archetype from 40.6% to 51.8%.

• Generalization to an external benchmark. On MathTutorBench, training raises the overall pedagogy score from 52.5% (untrained Qwen3-8B) to 79.2% (Sherpa DP), above PedagogicalRL's 63.2% and Sherpa NP's 73.4%. Instruction-following win rates rise from 74.0% to 90.3% in the standard setting and from 70.6% to 87.2% in the hard setting; scaffolding rises from 33.1% to 72.0% and from 32.4% to 67.3%.

• A reported side effect of teaching-specific training. Mistake-location accuracy declines across all trained teachers, and most also regress on Socratic questioning (Qwen3-8B: 25.3%, Sherpa DP: 21.9%). Sherpa DP nonetheless retains higher mistake-location accuracy than PedagogicalRL (30.3% versus 18.7%).

• Human teachers prefer the trained model. Across 320 dialogue contexts drawn from 80 CoMTA conversations labeled "Answer Not Accepted," 96 high school teachers recruited via Prolific gave Sherpa DP 728 judgments versus 187 for the base model, a 79.6% win rate excluding ties. Wins hold for both in-distribution preferences (subgoal decomposition 75.6%, contrastive comparison 85.5%) and out-of-distribution ones (causal justification 80.4%, error anticipation 76.4%). 87 of 96 teachers favored Sherpa DP more often across their judgments, and majority voting preferred it on 270 of 320 response pairs versus 34 for the base model.

• Prompted student archetypes are not selective or informative enough. In a 6×6 matrix pairing a GPT-5.6-Luna teacher using one of six fixed teaching strategies with students holding one of six preferences, prompted students showed neither a clear diagonal advantage in improvement nor consistently lower complaint rates along the diagonal, while the gated archetypes showed both.

• Predefined gate penalties hurt. Increasing gate-failure penalties reduces MathTutorBench Pedagogy Average from 79.2% (no penalty) to 70.6% (penalty 0.25) and 66.0% (penalty 0.5), and a larger penalty also reduces out-of-distribution generalization.

• Human preferences are not unanimous. A minority of teachers preferred the base response when they found Sherpa's explanations potentially overwhelming, which the authors note suggests adaptivity can occasionally come at the cost of simplicity.

Methodology in Plain English

The framework runs each teaching episode as a three-stage cycle. In preparation, the teacher privately works out a solution draft to keep as background knowledge, which helps small models avoid drifting as the dialogue grows. In tutoring, the teacher alternates private reasoning with a student-visible instruction across multiple turns. In testing, the student attempts the original problem with its accumulated context, and the same frozen student also attempts it with no interaction history; the reward is the difference in measured accuracy between the two.

The student is deliberately a weak model, Qwen3-1.7B, that cannot solve the problems unaided, so any gain is attributable to tutoring. Because such a model will not reliably behave differently just from a prompt, the authors use gates. The guidance gate has an LLM checker detect whether an instruction discloses the final answer; if so, the instruction is blocked. The adaptive gate checks whether the instruction matches the student archetype's assigned preference among six pedagogical strategies taken from education research: attempt diagnosis, causal justification, subgoal decomposition, step demonstration, independent verification, and contrastive comparison. A turn that fails either gate never enters the student's context; instead the student returns a scripted complaint such as "Don't directly show me the answer," which the teacher can condition on at the next turn.

For optimization, the teacher's per-turn advantage is the trajectory's return masked by gate acceptance, centered leave-one-out within a group of trajectories sharing the same problem and archetype, then normalized with local format and repetition penalties added afterward. Episodes where a trajectory is the only one reaching a given turn receive no leave-one-out comparison and only local penalties. The teacher is trained with asynchronous PPO using AReaL with decoupled PPO to handle rollouts from slightly older policies.

Seven archetypes are instantiated: six with distinct preferences plus one with no preference. Four (None, Attempt, Subgoal, Contrast) are in-distribution; three (Causal, Step, Verification) are held out. Problems come from MATH, filtered to remove those beyond the teacher's capability or already solvable by the student, leaving 759 training and 528 evaluation problems.

Why This Matters

The paper reframes teacher training around a measurable outcome, student learning gain, rather than an external rubric of good teaching. That shift matters because rubrics do not capture learner heterogeneity, and the results show that predefined penalties actively harm generalization on held-out pedagogical benchmarks.

• Adaptive tutoring systems. A trained model can infer what a learner needs from the conversation itself, rather than applying one fixed explanation style to everyone.
• Teacher-facing tools. The authors cite instructional suggestions for human tutors as an adjacent line of work; a model that detects when a student needs a subgoal or a comparison could support human instruction similarly.
• Curriculum and problem selection. The filtering step, discarding problems the student can already solve or the teacher cannot help with, is directly reusable for building practice sets with room for growth.
• Low-resource deployment. Because only the teacher is trained and the student remains a frozen small model, the approach targets a comparatively cheap 8B teacher rather than frontier-scale systems.

Industry relevance: The work sits in the competitive space of AI tutoring and educational LLM products, where vendors must demonstrate learning outcomes, not just answer accuracy. Prior evidence cited in the paper (Srinivasa et al., 2025) that none of 16 evaluated frontier models scores above 56% against expert-written tutoring rubrics indicates that raw problem-solving skill does not translate into teaching quality, leaving room for specialized post-training. Code and model are released at github.com/SALT-NLP/Sherpa.

Future Directions

• Richer and compositional student models. The teacher-student separation lets archetypes be refined independently of teacher training, and the authors suggest combinations of archetypes or scenarios beyond them.
• Students who do not know their own preferences. The paper raises the case where learners cannot state what helps them, requiring the teacher to infer preferences gradually through interaction rather than receiving any signal.
• Closing the regressions. Mistake-location accuracy falls for every trained teacher and most regress on Socratic questioning, indicating a training tradeoff that is not resolved.
• From simulated to real learners. All training uses language-model students, and the authors position human-teacher alignment as a proxy for the eventual goal of supporting real students in real-world settings.

Target Audience

Researchers and practitioners working on LLM post-training, reinforcement learning from interaction, and AI in education. It is most useful for those interested in outcome-grounded reward design, multi-turn RL, or learner simulation, and for teams building tutoring products who want to see how training signals can be grounded in measured student improvement rather than expert-authored rubrics. Readers without RL background will need to consult the appendices for the optimization details.

Authors’ abstract

Large language models (LLMs) have become increasingly capable problem solvers, but being able to solve a problem is not the same as being able to teach it. Existing approaches to training LLMs as teachers rely on demonstrations, preference data, or predefined pedagogical criteria that specify what good teaching looks like. However, these signals are often not grounded in individual student learning outcomes, where effective teaching strategies can vary substantially across learners. To address this, we introduce Sherpa, a multi-turn reinforcement learning framework that instantiates multiple student archetypes with LLMs conditioned on distinct learning preferences and trains a teacher model to adapt its instruction by directly maximizing their learning outcomes. Teacher LLMs trained with Sherpa improve instructed students' performance across all archetypes by an average of 20.5 percentage points. Under MathTutorBench's evaluation, Sherpa raises the overall pedagogy score from 52.5% to 79.2%, indicating better teaching responses. Our human studies show that the trained teacher is preferred over the base model in 79.6% of pairwise comparisons. Together, Sherpa trains LLM teachers to adapt to diverse simulated students and become better aligned with human teachers, paving the road towards AI tutors teaching real students.