Learning Engagement Assistant (LEA): Cross-Course Scalability and Classroom Evaluation of an Agentic AI Tutoring System

AI summary

Overview

Research area: AI in education — specifically agentic Intelligent Tutoring Systems (ITS), Retrieval-Augmented Generation (RAG) for course grounding, Knowledge Component (KC) modelling, and human-subjects evaluation of AI tutors.

Technical level: Intermediate. The paper assumes familiarity with RAG pipelines, LLM agents, and educational-psychology constructs such as Cognitive Load Theory and the Zone of Proximal Development, but explains its architecture and evaluation metrics in accessible terms.

Scope: An extended conference paper reporting the first real-classroom deployment and the first empirical cross-course scalability test of LEA, an agentic AI tutoring system previously validated only through simulation.

What This Paper Is About

LEA is an AI tutoring system that combines three modes (Chat, Tutor, Quiz) with a course-specific RAG knowledge base, a hierarchical Knowledge Component model, and an agent that continually estimates a learner's cognitive load, Zone of Proximal Development, and motivational state. A prior ICAART 2026 paper validated it purely through simulation on a single graduate STEM course (CMP511), using 14,143 synthetic interactions across six synthetic learner profiles, and left two questions open: does it work with real students in a real classroom, and does it scale to other courses without architectural changes? This paper answers both by deploying LEA with real students (n = 8, CMP511) and across three courses spanning two academic levels and two disciplinary domains, comparing what was observed against what the simulation had predicted.

Key Contributions

1. Production deployment across three university courses at Abertay University, Dundee: CMP511 (Machine Learning and Artificial Intelligence, MSc), CMP202 (Data Structures and Algorithms 2, undergraduate), and PSY555 (Human Psychology, MSc). This spans two degree levels and two disciplinary domains, with PSY555 — a social-science course with discursive, non-procedural content and zero code-based Granular Objectives — included as a cross-domain stress test. It provides the first empirical account of LEA's scalability claim: that the orchestration layer generalises while the knowledge layer adapts.

2. The first pilot classroom study of LEA with real students enrolled in CMP511 (n = 8), conducted under ethics approval with pseudonymised data collection, using a nine-section survey instrument covering usability, mode-specific perceptions, learning outcomes, trust, and technical reliability.

3. A cross-course scalability evaluation of 660 questions across all three courses using the RAGAS framework to measure retrieval and generation quality.

4. Documentation of a course-blind failure mode: the Quiz mode's question-generation logic hard-coded Python, causing it to generate Python code questions for PSY555 (which has no programming content) and Python instead of C++ for CMP202. This was the only cross-course failure requiring a code change; it was resolved with an explicit course-to-programming-language mapping.

Main Findings

• Simulation did not fully predict classroom behaviour. The classroom deployment revealed a divergence from simulation predictions across modes, leading the authors to state that synthetic evaluation alone cannot anticipate all aspects of real deployment.

• Retrieval and relevancy were broadly stable across courses. In the 660-question RAGAS evaluation, Answer Relevancy ranged from 0.88 to 0.94 and Context Precision from 0.88 to 0.90 across courses.

• Faithfulness declined with curriculum distance. Faithfulness fell from 0.69 to 0.50 as courses moved away from the system's original course. The authors frame this as a preliminary finding that may reflect generation logic tuned to the system's original subject rather than a scalability limitation.

• The orchestration layer required no modification. The Agent Orchestrator, Mastery Tracker, and scaffolding/motivation engines operate on abstract KC identifiers and learner-state variables and contain no course-specific content. Onboarding each of the three courses required only populating two artifacts: a ChromaDB RAG vector store and a hierarchical JSON KC model. The single exception was the course-blind code-question generator described above.

• Course profiles differ substantially. CMP511 (MSc, Computing, 11 weeks) has 48 Learning Objectives, 161 Granular Objectives, 15 (9.3%) code-based GOs, 90 RAG documents and 1,266 chunks. CMP202 (UG, Computing, 10 weeks) has 30 LOs, 90 GOs, 32 (35.6%) code-based GOs, 112 documents and 1,230 chunks. PSY555 (MSc, Social Science, 13 weeks) has 16 LOs, 79 GOs, 0 (0%) code-based GOs, 64 documents and 691 chunks.

• Context Precision and Context Recall are interpreted with a caveat. In the absence of human-authored ground truth, the system's own generated answer was used as a stand-in reference, so these two metrics are treated as measures of internal retrieval-generation consistency rather than absolute retrieval quality.

• Prior simulation baselines (from the conference paper). Chat mode: n = 1,980, 100% retrieval success, M = 0.880 generated answer quality (95% CI [0.879, 0.882]). Tutor mode: n = 5,563 turns, Multi-Turn Effectiveness M = 0.861 (95% CI [0.859, 0.863]), Adaptive Feedback Appropriateness M = 0.807 (95% CI [0.805, 0.809]). Quiz mode: n = 6,600 items. A two-way ANOVA on the Learner State Match metric found learner profile, not instructional week, to be the dominant factor — this sentence is truncated in the available content, so the full result is not reported here.

• The wider evidence base is thin. General-purpose LLM learning interfaces (ChatGPT Study Mode, Gemini Learn LM, Claude for Education) and purpose-built tutors such as Khanmigo and Harvard's CS50 AI Tutor lack a unified combination of KC modelling, mastery tracking, course-specific RAG, and cognitive load / ZPD / motivational state estimation. The largest reported classroom evidence cited is a meta-analysis of 35 experimental studies (4,193 participants) reporting Hedges' g = 0.670 for ChatGPT on learning outcomes. A 2026 systematic review of 86 tutoring-system studies found only 9% addressed social-science content versus 52% in STEM.

• Specific classroom survey results are not reported in the available content. The paper states that the nine-section instrument was administered and that a direct comparison between simulation predictions and observed student experience appears in Section 7, but the individual survey outcomes are not included in the portion of the paper provided.

Methodology in Plain English

The work has three strands.

Architecture and course onboarding. LEA separates "what the system does" from "what it knows." The orchestration logic — cognitive load estimation, ZPD tracking, motivation classification, scaffolding and fading — works on abstract objective identifiers and learner-state variables, not on course content. Adding a course means building two artifacts: a RAG library and a KC model. The RAG library is built by validating files via MD5 signature comparison, converting PDFs with PDFMiner, videos with Whisper, and Office documents with MarkItDown, summarising with GPT-4, chunking semantically into 500-token windows with 100-token overlap bounded at sentence level with NLTK, and embedding with OpenAI's text-embedding-3-small into a dedicated ChromaDB collection. The KC model is authored by an instructor in an Excel template; a script decomposes Learning Objectives into Granular Objectives, assigns Bloom's-taxonomy levels, sets mastery thresholds, and identifies cross-week dependencies; the instructor then reviews and edits the result before it becomes the final KC model file. The same three-stage pipeline was applied identically to all three courses, with the only variation coming from the source materials and instructor-authored objectives.

Classroom pilot. Eight CMP511 students used LEA under ethics approval with pseudonymised data collection and completed a nine-section survey on usability, mode-specific perceptions, learning outcomes, trust, and technical reliability. These responses were compared directly against the predictions established by the earlier simulation.

Cross-course scalability evaluation. A set of 660 questions was run across all three courses and scored with the four RAGAS metrics: Faithfulness (whether generated claims are supported by retrieved context), Answer Relevancy (whether the answer addresses the question), Context Precision (signal-to-noise of retrieved chunks, weighted by rank), and Context Recall (completeness of retrieved context). Faithfulness and Answer Relevancy are reference-free; the other two normally need a reference answer, for which the system's own output was substituted.

Why This Matters

Impact on research. The paper moves evaluation of AI tutors past simulation-only validation and past single-course demonstrations. It shows empirically what happens when a course-agnostic adaptive architecture meets a level change (postgraduate to undergraduate) and a domain change (computing to social science), and it isolates exactly which layer survived unchanged (orchestration) and which did not (code-question generation). It also contributes a concrete testing protocol — RAGAS across courses with a distance-from-original-course framing — to a field the paper describes as lacking a standard methodology for evaluating cross-course generalisation.

Real-world applications:

• Universities wanting one adaptive tutoring backend to serve multiple departments without rebuilding the adaptive engine per course.
• Onboarding any new module by supplying course materials and an instructor-authored Knowledge Component spreadsheet, rather than new code.
• Grounding LLM tutoring responses in validated course materials so answers stay aligned with the taught curriculum rather than general model knowledge.
• Formative assessment generation that respects the course's actual programming language (or generates none, as with PSY555).
• Converting heterogeneous teaching materials — slides, lecture recordings, tutorial notes, code files — into a searchable vector store for student-facing Q&A.

Industry relevance. The findings matter to educational-technology vendors and institutions deploying vendor systems (Khanmigo, CS50's AI Tutor, ChatGPT Study Mode, Gemini Learn LM, Claude for Education) that the paper characterises as strong at content generation but weak on structured learner modelling. The specific failure mode — generation logic silently carrying an assumption from the original course into unrelated domains — is a general warning about hard-coded defaults in domain-agnostic claims, and the Faithfulness decline suggests that "domain-agnostic" pipelines may still need per-domain tuning of the generation layer.

Future Directions

• Full course-agnosticism of downstream components. The orchestration layer generalises, but the authors conclude that full course-agnosticism of all downstream components requires further investigation, particularly whether the Faithfulness decline reflects a genuine scalability limit or generation logic still tuned to the original subject.
• Calibration refinement. The cognitive-load coefficients were fixed via offline Monte Carlo simulation, and the paper explicitly leaves calibration open to refinement (Section 10).
• Standard methodology for cross-course evaluation. The paper notes that no standard methodology exists for evaluating cross-course generalisation of adaptive tutoring architectures, and the RAGAS-based approach used here is offered as a starting point rather than a settled answer.
• Broader and larger human-subjects evidence. The classroom study is described as a first pilot with n = 8 in one course; the paper situates it against a literature where human-subjects studies tend to be small-scale, short-term, or focused on a single interaction feature, and against a disciplinary imbalance (9% social science versus 52% STEM across 86 reviewed studies).

Target Audience

Researchers and graduate students working on intelligent tutoring systems, educational LLM agents, and RAG evaluation; learning designers and instructors evaluating course-grounded AI assistants; university teaching-and-learning staff who need to know what does and does not transfer when a tutoring system moves between departments; and educational-technology practitioners who must judge scalability claims that are frequently asserted but rarely tested empirically. Readers from AI safety, ethics, and human-subjects research will find the deployment methodology, ethics-approved classroom pilot, and trust measurement relevant, since these are the dimensions the paper argues that simulation cannot cover.

Authors’ abstract

This paper is an extension of a paper presented at the ICAART 2026 conference, which introduced LEA (Learning Engagement Assistant), an adaptive AI tutoring agent combining course-specific Retrieval-Augmented Generation (RAG) with structured Knowledge Component (KC) models across integrated Chat, Tutor, and Quiz modes. That prior work validated LEA on a single STEM course (CMP511) exclusively through simulation, using synthetic learner agents. This paper extends that work by reporting the first classroom deployment of LEA with real students (n = 8, CMP511) and the first empirical test of its cross-course scalability, deploying the system across three courses spanning two academic levels and two disciplinary domains. The study reveals a divergence from simulation predictions across modes, showing that synthetic evaluation alone cannot anticipate all aspects of real deployment. A RAGAS-based cross-course scalability evaluation (660 questions) finds Answer Relevancy and Context Precision broadly stable across courses (0.88-0.94 and 0.88-0.90 respectively), while Faithfulness declines with curriculum distance from the system's original course (0.69 to 0.50), a preliminary finding that may reflect generation logic tuned to the system's original subject rather than a scalability limitation. These findings suggest that while the orchestration layer requires no modification, full course-agnosticism of all downstream components requires further investigation.