Evidence-Decision-Feedback: Theory-Driven Adaptive Scaffolding for LLM Agents

AI summary

Overview

Research area: AI in Education (AIED), LLM-based pedagogical agents, multi-agent architectures

Technical level: Intermediate — readers should have basic familiarity with LLMs, agentic systems, and educational learning theory, but no deep technical background is required to follow the framework or findings.

Scope: The paper introduces Evidence-Decision-Feedback (EDF), a theoretical framework for building adaptive scaffolding into LLM pedagogical agents, and evaluates its instantiation (Copa) in an authentic high school physics classroom.

What This Paper Is About

LLM tutoring agents are often "one-size-fits-all," can encourage unhealthy student dependence, and produce feedback whose reasoning is opaque to teachers and students. The paper proposes EDF, a three-part framework that separates learner-state inference, pedagogical decision-making, and feedback generation so that LLM agents can scaffold students in open-ended learning environments in a way that is personalized, theoretically grounded, and interpretable. The authors build Copa, a collaborative peer agent for STEM+C problem-solving, and test it with 33 high-school sophomore dyads over six weeks.

Key Contributions

1. The EDF framework. A theory-linked architecture for adaptive scaffolding that decomposes agent behavior into three semi-autonomous modules — Evidence (learner-state inference via Evidence-Centered Design and Stealth Assessment), Decision (dialogue policy selection grounded in Social Cognitive Theory and the Zone of Proximal Development), and Feedback (adaptive talk moves grounded in Social Constructivism). Interpretability is treated as a core property, with each stage traceable through observable artifacts.

2. Copa, a multi-agent instantiation of EDF. A collaborative peer agent built on GPT-5 inside the C2STEM modeling environment, comprising four specialized sub-agents (StrategyAgent, AssessmentAgent, KnowledgeAgent, DialogueAgent) sharing a single learner model, each emitting chain-of-thought "evidence" to justify its output.

3. A classroom-scale empirical evaluation. An authentic deployment with n = 33 high-school dyads producing 7,017 logged environment actions and 238 student-agent turns, analyzed across four research questions: adaptivity, understanding-mastery alignment, student reliance, and interpretability.

4. The "helpfulness paradox" concept. A named design tension: pedagogical agents must be helpful enough to earn students' trust and use, yet restrained enough to avoid cognitive offloading that undermines durable learning.

Main Findings

• Scaffolding adapts as mastery grows. Copa significantly reduced its PROBE_UNDERSTANDING policy as task mastery increased (Spearman's ρ = −0.34, p = 0.034) while increasing SUGGEST_ACTION (ρ = 0.33, p = 0.039) and PUSH_LIMIT (ρ = 0.42, p = 0.007). This matches ZPD principles: Copa shifts from assessing understanding to supporting application and extension.

• Understanding and mastery move together. Students' success rate at correctly explaining their code after a probe rose significantly with task mastery (ρ = 0.40), suggesting students were not simply "gaming the system" — mastery gains tracked verbalizable conceptual understanding.

• Students rely less on Copa as they improve. Requests for agent support dropped significantly as mastery increased (ρ = −0.26), with roughly 59% of all support occurring at mastery below 40%. This is consistent with developing self-regulation and self-efficacy.

• Agent reasoning is traceable at every stage. All three EDF links showed statistically significant non-random structure (p < 0.001): Grounding (data → evidence) achieved keyword recall of 0.43 vs. 0.21 random; Alignment (evidence → decision) achieved SBERT similarity of 0.64 vs. 0.39; Faithfulness (decision → feedback) achieved 0.48 vs. 0.24.

• Students valued inquiry but disliked being denied direct answers. Survey ratings were positive for "questions were appropriate" (3.81/5) and "made me think about my model" (3.88/5), but lower for perceived understanding (2.69) and feedback usefulness (2.77). Students and teachers both noted frustration when immediate answers were withheld.

• Multi-lens evaluation matters. The authors argue that adaptivity, learning alignment, reliance, interpretability, and stakeholder perception are complementary evaluation axes; optimizing one can mask failure in others.

Methodology in Plain English

The research revolved around a multi-year participatory design process with students, teachers, and learning scientists, synthesized into a three-module framework. The authors then built Copa to instantiate each module concretely.

Copa is a "knowledgeable peer" persona, deliberately chosen (by students themselves) over an expert-tutor role to reduce dependency. It runs inside C2STEM, a block-based computational modeling environment where students simulate 1-D and 2-D kinematics. As students work, C2STEM logs their actions (e.g., editing a code block, setting a velocity variable). A background team of three asynchronous agents converts those raw logs into a coherent learner model: a StrategyAgent identifies problem-solving patterns (e.g., systematic tinkering vs. haphazard building), an AssessmentAgent triangulates the current state against past states to produce an updated learner state, and a KnowledgeAgent compares the current model to an expert reference and retrieves ZPD-aligned domain knowledge. A fourth agent, the DialogueAgent, is the orchestrator: it runs synchronously with the student to decide the pedagogical intent (a dialogue policy, such as PROBE_UNDERSTANDING, SUGGEST_ACTION, or PUSH_LIMIT) and produce a specific talk move.

Prompts were engineered with CoTAL (Chain-of-Thought Prompting + Active Learning), a human-in-the-loop method using chain-of-thought reasoning and few-shot exemplars refined across four hour-long sessions with 10–15 researchers. GPT-5 was selected after testing Gemini, Claude, and GPT variants.

Evaluation combined quantitative analysis of log and dialogue data with statistical tests (Spearman's ρ for ordinal mastery data, permutation tests for interpretability metrics), survey responses, and semi-structured debriefs with three participating educators. All findings were manually triangulated against student conversations and screen recordings, with IRB approval and anonymization.

Why This Matters

Research impact

The paper bridges a long-standing gap between intelligent tutoring systems (which have strong theory and interpretability but weak dialogue flexibility) and LLM-based agents (which have rich dialogue but poor pedagogical grounding). EDF provides a concrete template for making LLM agent behavior traceable to learning theory, and the authors argue for multi-dimensional — rather than single-metric — evaluation of pedagogical agents.

Real-world applications

• K-12 STEM classrooms: Deployable peer agents that scaffold rather than solve, reducing overreliance on AI during formative learning.
• Open-ended learning environments (OELEs): Inquiry and modeling environments (physics, computational thinking, engineering design) where nonlinear student progress defeats rule-based tutors.
• Teacher-facing dashboards: Interpretable evidence chains give teachers visibility into what students understood and why an agent acted as it did.
• Educational policy and procurement: Interpretability and theory-traceability provide the audit trail that schools and districts need before adopting LLM systems.

Industry relevance

EdTech vendors building LLM tutors face a trust and efficacy problem: tools that inflate task performance without building transferable skill are being scrutinized. EDF offers a modular architecture that separates inference, policy, and generation — a pattern that maps cleanly onto production multi-agent pipelines. The interpretability metrics (grounding, alignment, faithfulness) offer an evaluation vocabulary that could become standard for compliance and quality assurance in educational AI.

Future Directions

• Broaden the adaptation signal. The current study only adapts on task mastery. Future work should incorporate self-regulation, self-efficacy, social dynamics, and other learner attributes into the policy decision.
• Run a randomized controlled trial. Findings are correlational; institutional constraints prevented a causal comparison against alternatives or against no-agent conditions.
• Generalize beyond Copa. EDF is presented as a general framework, but has only been instantiated and evaluated through one agent in one curriculum. Testing across other subjects, age groups, and agents is needed.
• Resolve the helpfulness paradox empirically. How to design agents that earn student trust while still resisting the pull toward answer-giving remains an open design and measurement question, tied to student buy-in and expectations shaped by prior ChatGPT use.

Target Audience

AIED and learning-science researchers; developers of LLM-based pedagogical agents and multi-agent architectures; educators, curriculum designers, and school administrators evaluating AI tutoring tools; and industry teams building trustworthy educational AI who want a theory-grounded alternative to end-to-end prompt-only tutoring systems.

Authors’ abstract

LLMs offer tremendous opportunities for pedagogical agents to help students construct knowledge and develop problem-solving skills, yet many of these agents operate on a "one-size-fits-all" basis, limiting their ability to personalize support. To address this, we introduce Evidence-Decision-Feedback (EDF), a theoretical framework for adaptive scaffolding with LLM agents. EDF integrates elements of intelligent tutoring systems (ITS) and agentic behavior by organizing interactions around evidentiary inference, pedagogical decision-making, and adaptive feedback. We instantiate EDF through Copa, a Collaborative Peer Agent for STEM+C problem-solving. In an authentic high school classroom study, we show that EDF-guided interactions align feedback with students' demonstrated understanding and task mastery; promote scaffold fading; and support interpretable, evidence-grounded explanations without fostering overreliance.