Cite or Decline: A Strict Course-Grounded Chatbot for STEM Lecture Videos

AI summary

Overview

Research area: Educational natural language processing, specifically retrieval-augmented generation (RAG) applied to STEM lecture-video question answering, combined with a real classroom deployment study.

Technical level: Intermediate. The paper describes a RAG pipeline with retrieval metrics (MRR, nDCG@5, P@5, hit rates) and a weighted audit procedure, but it is written around deployment behavior and student perception rather than novel model architecture.

Scope (one sentence): A two-phase deployment of chapter summaries and a course-isolated, citation-first chatbot on the VideoPoints lecture-video platform, evaluated through production logs, surveys, a diagnostic audit, and a controlled retrieval benchmark on EduVidQA.

What This Paper Is About

Indexed lecture videos let students jump to topics, but they cannot answer a student's specific question, and a general-purpose chatbot answers in language that may not match the instructor's framing or the actual course material. The authors built and deployed a chatbot that answers only from the active course's lecture materials and returns clickable timestamped citations, then studied how students actually used it over a semester. They also tested whether the retrieval design behind it holds up on a public lecture-video question-answering benchmark.

Key Contributions

1. A semester-long, multi-course deployment study characterizing how students used and perceived a citation-first lecture-video chatbot across Phase 1 (Fall 2025, summaries only, 128 survey respondents) and Phase 2 (Spring 2026, chatbot live, 41 survey respondents including 22 self-reported chatbot users, 833 messages).
2. Separate evaluation of grounding and retrieval design. A production audit estimates how often cited lecture segments support the generated answer, while a controlled offline benchmark compares the retrieval design against dense-only and course-unrestricted baselines.
3. A diagnostic audit protocol over 224 unique production messages, with stratified sampling weights mapped back to the 833-message population, examining citation support, evidence relevance, refusal appropriateness, and implicit refusals missed by regex.
4. Transferable deployment lessons about course isolation, question reformulation, and the mismatch between retrieval-based question answering and students' task-oriented requests. The authors state explicitly that the contribution is not a new RAG architecture.

Main Findings

• Course isolation held completely in production. Of 833 messages, 587 carried citations (70.5% coverage, 95% CI 67.3–73.5). Each cited message displayed exactly seven citations, a fixed output cutoff, producing 4,109 citation events. None pointed outside the active course (0/4,109).

• Citations did not always support the answers. The weighted diagnostic audit estimated 65.0% of cited non-refusal answers were fully supported (95% CI 55.0–75.0) and 86.3% were fully or partially supported (78.8–93.8). An estimated 11.3% were unsupported (5.0–18.8), and irrelevant-evidence citations were estimated at 1.7% (0.001–0.044, n=118).

• Refusal frequency did not mean refusal quality. Of 246 uncited messages, 202 contained an explicit refusal detected by regex, but 27 of the remaining 44 contained implicit refusal language the regex missed. The audit estimated 61.9% of refusals were appropriate (51.4–71.8). A separate audit of 50 uncited messages labeled 54% of queries clearly unanswerable from available course evidence and 46% unclear, rating 58% of refusals appropriate, 38% unclear, and 4% inappropriate.

• Imperative commands fared worse than questions. Questions and keyword fragments together accounted for 688 of 833 messages (82.6%) with similar refusal and no-citation rates. Imperative messages had higher odds of refusal than non-imperative messages (odds ratio 2.74, 95% CI 1.55–4.86, Fisher's exact p = 0.0007) and higher odds of returning no citation (odds ratio 3.74, 95% CI 2.10–6.68, p < 10⁻⁵).

• Agentic expectations were the sharpest boundary. At least 61 messages (7.3%) expressed agentic expectations such as generating practice questions, grading, or remembering previous turns. These had a 78.7% refusal rate and an 86.9% no-citation rate, versus 25.0% for both outcomes among other messages.

• Course isolation was the single largest retrieval lever. On the real-world split of EduVidQA (269 questions, k = 5), removing course isolation dropped video hit from 0.747 to 0.569, a decrease of 17.8 percentage points, statistically supported for video hit (p ≤ 0.0005) and timestamp hit (p ≤ 0.034). The unrestricted system still retrieved above-threshold evidence for 98.9% of questions, but often from the wrong course.

• The full retrieval design modestly beat dense-only. Full design scored timestamp hit 0.457, video hit 0.747, P@5 0.129, MRR 0.293, nDCG@5 0.330, versus dense-only at 0.402, 0.684, 0.103, 0.250, 0.283. This is a 6.3 percentage point improvement in correct-lecture retrieval over dense-only. Paired-test p-values ranged from 0.010 to 0.032, with small effect sizes (d_z = 0.14 for MRR, d_z = 0.16 for nDCG@5).

• BM25 carried most of the timestamp-level gain; the summary prior was a refinement. Adding BM25 moved timestamp hit from 0.402 to 0.442; adding the summary prior moved it further to 0.457. The authors report no paired significance test for the BM25-versus-full comparison and treat the prior as a ranking refinement rather than a statistical claim.

• Usage concentrated and aligned with assessment periods. The most active course produced 656 of 833 messages (78.8%). The March 23–27 window contained 328 messages (39.4%), within the university's midterm dates of March 23–31, and May 4–11 overlapped finals dates of May 6–12. The longest session contained 68 turns. Logs recorded activity on 60 days.

• Empty sessions inflate raw traffic. Of 680 sessions, 447 contained no stored exchange; 233 sessions had at least one message. Mean/median/max active session depth was 3.58 / 2 / 68, with 108 sessions having one message and 38 having at least five. Students clicked timestamped citations 313 times (37.6%).

• Students rated summaries and citations highly, but wanted more. Phase 1 item means ranged from 4.34 (perceived summary accuracy, 81.4% top-2-box) to 4.67 (chapters help find content, 92.0%). Phase 2 among 22 users: ease of use 4.64 (95.5%), citation links helpful 4.55 (100.0% top-2-box), exam/quiz helpfulness 4.55 (86.4%), trust in accuracy 4.27 (90.9%). Practice-question generation and grading was rated 4.59 (86.4%) as the strongest unmet need.

• Summary length preference was task-dependent. For navigation, 75 of 112 respondents preferred the existing length of about five sentences. For review, only 48 of 113 preferred the existing length while 61 wanted longer or much longer summaries.

• Some errors traced to noisy evidence, and reformulation sometimes worked. Case analysis identified short or misspelled questions, broad multi-chapter requests, generation requests, and requests for conversational memory as recurring difficulties, and confirmed at least one cited response that was incorrect because the underlying lecture evidence was noisy. Of 170 no-citation turns followed by another turn, 93 were followed by a citation-present response, consistent with reformulation.

Methodology in Plain English

The system was built on VideoPoints, a production lecture-video platform at a large public university, using its existing infrastructure so the only marginal cost was model API usage (approximately $100 total including development, with answer generation about $55 and summary generation about $45).

Chapter segmentation used Gemini-2.5 flash-lite and chapter titles and summaries used GPT-4.1 Mini. In Phase 1 (Fall 2025), summaries were student-facing only; in Phase 2 (Spring 2026), those same summaries became a retrieval signal.

When a student asks a question, the system encodes it with all-MiniLM-L6-v2 embeddings, restricts retrieval to the active course, and ranks transcript and slide-text chunks using a combined score: dense cosine similarity (weight 1.0), BM25 lexical relevance (weight 1.25), and a chapter-summary prior (weight 0.4). The prior is a soft signal, not a hard filter — every chunk stays eligible, but chunks from chapters whose summaries match the question better get a ranking boost. Each summary bullet is embedded separately and a chapter's prior is the maximum query-to-bullet similarity. Two edge cases preserve the candidate pool: 19 chapters with failed summary validation were dropped from the prior index while their chunks stayed eligible, and 100 chunks whose chapters had no usable summary inherited the per-query minimum prior rather than zero, because cosine similarity can be negative. The top-k chunks go to Gemini-2.5 flash-lite for answer generation with a prompt requiring timestamped grounding, and the output is an answer with citations in the format [SOURCE n, HH:MM:SS].

For evaluation, the authors combined production logs, retrieval traces, surveys (optional, IRB-approved, 1–5 Likert scales), and the diagnostic audit. Session identifiers were salted and hashed before analysis, raw logs stayed on local infrastructure, and questions and responses were de-identified and screened for direct identifiers before being sent to an external model API. The audit used nvidia/nemotron-3-ultra-550b-a55b with temperature 0.0 and a fixed seed; of 1,107 records attempted, 1,106 parsed successfully. The audit censused the two small citation-by-refusal cells (39 cited refusals, 44 uncited non-refusals) and all 51 imperative messages at weight 1.0, and sampled 60 of 548 cited non-refusals (weight 9.13) and 40 of 202 uncited refusals (weight 5.05), giving 234 selections and 224 unique messages after deduplication.

Because production traffic could not provide a course-unrestricted baseline without violating the instructors' requirement, the retrieval design was tested on the real-world split of EduVidQA, a public multimodal lecture-video QA benchmark. That evaluation uses 269 questions from 99 lectures spanning seven of ten courses, within a corpus of 139 videos, 10 courses, 1,062 chapters, and 5,924 chunks. An LLM grouped videos into courses and generated chapter boundaries and summaries once, reused unchanged across all retrieval arms. Every arm used the same corpus, questions, embeddings, chunking, context budget, and retrieval budget.

Why This Matters

Impact on research. The paper argues that course isolation, citation presence, citation support, and refusal appropriateness are four separate properties that should be evaluated separately — a citation in an answer does not prove the cited segment supports it, and a frequent refusal does not prove each refusal was appropriate. It also shows that confidence-style retrieval metrics can reward exactly the behavior instructors prohibit: without course isolation, the system still surfaced above-threshold evidence for 98.9% of questions, often from the wrong course. The study provides a deployment-grounded counterpoint to benchmark-only evaluations of educational RAG.

Real-world applications:

• University lecture-video platforms that want a study companion constrained to the instructor's own materials, with clickable timestamped navigation back into the recording.
• Instructor-controlled assessment support, since the strongest unmet student request was practice-question generation and grading (rated 4.59/5), a task the retrieval-only design repeatedly declined.
• Academic-integrity policy design, because some students pasted multiple-choice or true-false items into the chatbot, prompting the authors to recommend assessment-mode policies and explain-without-answer options.
• Deployment cost planning for teaching budgets, since the entire two-phase operation cost approximately $100 in model API usage on existing lab infrastructure.

Industry relevance. The finding that a light lexical signal (BM25, weight 1.25, above the dense weight of 1.0) produced most of the timestamp-level retrieval gain, while a summary-based prior added a small additional increase, is a practical data point for teams building grounded assistants over noisy conversational content. The result that scope restriction mattered more than either scoring refinement is directly relevant to any enterprise RAG deployment where correct-source attribution is a hard requirement rather than a preference.

Future Directions

• Human validation of citation support and refusal appropriateness. The current audit uses automated judgments from a different model family than the generator, and the authors state they still do not report a human-validated factuality rate. They recommend following the automated audit with human evaluation.
• Instructor-controlled task tools. Students repeatedly requested practice-question generation, grading, planning, and conversational memory — requests the system refused 78.7% of the time. The authors call for clear boundaries and instructor-controlled tools for tasks beyond grounded question answering.
• Resolving the answerable-but-missed question. The audit's course-content inventory was assembled only from passages cited elsewhere in the same course, so material retrieval never surfaced was invisible to the judge; the authors therefore decline to report an answerable-but-missed population estimate, which they note would be biased toward zero.
• Instrumenting latency and user-level metrics. Response latency was not instrumented during the deployment window, so speed is reported as a design constraint rather than a measured property, and no user identifiers or enrollment denominators were available, so unique-user counts and adoption rates cannot be computed.

Target Audience

Researchers and practitioners working on educational NLP, retrieval-augmented generation, and conversational systems grounded in private document collections. It is also relevant to instructors and instructional-technology teams deciding whether to deploy AI study tools in real courses, and to engineers who need concrete evidence about how retrieval scope, lexical signals, and summary-based priors affect grounding quality. Readers looking for a new RAG architecture will not find one here; readers looking for deployment evidence under real instructional constraints will.

Authors’ abstract

Recorded lecture videos, often enhanced with search and summarization features, are a standard study resource. However, students cannot easily ask course specific questions or verify answers against an instructor's lecture. We report a semester-long deployment of VideoPoints platform with a retrieval-augmented chatbot that answers from course lecture materials and returns timestamped citations. The chatbot retrieves only from the active course, uses chapter summaries to guide transcript ranking, and returns clickable timestamped citations. Students used it for quick lookups and exam review. Across 833 messages, 70.5% included citations, none crossed a course boundary, and when no lecture evidence matched, the chatbot usually declined rather than answering. Among the users, citations were the most consistently useful feature, while practice-question generation was the strongest unmet request. We also evaluated the design on the real-world test split of EduVidQA, a public multimodal benchmark for lecture-video question answering. Our design improved correct-lecture retrieval by 6.3 percentage points over dense-only retrieval. Together, the results show that effective deployment depends on course isolation, supported citations, and alignment with students' study practices.