Stan: An LLM-based thermodynamics course assistant

AI summary

Overview

Research area: Applied natural language processing and AI-in-education, specifically retrieval-augmented generation (RAG), local/open-weight LLM deployment, and automatic speech recognition applied to an undergraduate chemical engineering thermodynamics course (CHEG231, Chemical Engineering Thermodynamics I, University of Delaware; Fall 2025 offering).

Technical level: Intermediate. The paper assumes familiarity with LLMs, RAG pipelines, prompt design, and speech-to-text transcription, but explains its design choices in practical rather than mathematical terms.

Scope in one sentence: The paper describes the design, implementation, and observed failure modes of Stan, a locally hosted suite of dual-purpose (student-facing and instructor-facing) LLM tools built on lecture transcripts and a structured textbook index for a thermodynamics course.

What This Paper Is About

Most discussion of AI in education centers on student-facing tools such as chatbots, tutors, and problem generators, while the same underlying infrastructure for supporting instructors remains largely unexplored. This paper builds and deploys a shared data foundation of lecture transcripts and a structured textbook index in two roles at once: a student-facing textbook query tool that returns grounded answers with chapter and page references, and an instructor-facing lecture analysis tool that produces per-lecture summaries, extracts questions, flags confusion, and catalogs anecdotes and analogies. The goal is a system that runs entirely on locally controlled hardware with open-weight models, avoiding cloud APIs while providing both a study resource and a semester-scale, searchable record of teaching.

Key Contributions

1. A dual-role data pipeline. A single infrastructure built from lecture transcripts and two structured textbook representations (a hierarchical index of approximately 1,500 entries, bindex_tab.json, and a table-of-contents navigation tree, ftoc_nav_tree.json) serves both students and instructors from the same corpus.

2. A student-facing RAG query pipeline with a transparent dual-path term extractor. Student queries are processed in parallel by a deterministic regex extractor and a local LLM extractor (Llama 3.1 8B via Ollama), merged with a max-score strategy, matched against the textbook index, and synthesized into a 3–5 sentence grounded response with chapter and page citations.

3. An instructor-facing lecture analysis pipeline with four analysis types. Each transcript is processed into structured JSON for summary, questions, confusion detection, and anecdotes/analogies, producing a searchable record that supports course reflection, reminders, and improvement.

4. A documented account of practical failure modes and mitigations for 7–8 billion parameter models. The paper reports context truncation, bimodal output distributions, schema drift, Whisper hallucination loops, and domain-term misrecognition, along with the specific mitigations that resolved them.

Main Findings

• Domain vocabulary prompting substantially improved technical transcription. Without prompting, "fugacity" was consistently rendered as "GASB," "Antoine equation" as "Anaconda equation," and "Peng-Robinson equation of state" as "time-Robinson." Supplying thermodynamic terms through Whisper's initial_prompt parameter corrected "fugacity" throughout the transcript, though some terms such as Peng-Robinson remained occasionally garbled until the prompt was expanded to the full term "Peng-Robinson equation of state."

• A three-layer hallucination strategy reduced repetition loops from 6.4% to 0.02% of segments. The baseline single lecture contained four loops: "Elizabeth." repeated 37 times (50.5 s), "That's a lot." ×10 (10.2 s), "You told me to." ×3 (10.6 s), and "Okay." ×3 (3.0 s), totaling 53 of 826 raw segments (6.4% by count, 2.2% by time). Across the full corpus of 39 lectures with all mitigations active, only one loop was detected ("What?" ×5 in lecture 30) and the post-processing filter removed 4 segments totaling 5 seconds.

• Transcription throughput was 49× realtime for the full semester. The full corpus of 39 lectures (35.7 hours of audio) was transcribed in 43.7 minutes of wall-clock time on an NVIDIA RTX 4090 (24 GB VRAM), consuming approximately 4.3 GB of VRAM, with individual lectures ranging from 37× to 56× realtime. The baseline single lecture transcribed in 177 s (18.6×).

• Institutional and domain-prompted transcripts were similar in aggregate but differed in contextual fidelity. Across six matched lecture pairs (lectures 26–31), mean word counts were 7,449 (Kaltura) versus 7,409 (Whisper), a ratio of 0.99–1.01, and high-frequency domain terms appeared at nearly identical rates (entropy: 329 vs 331; enthalpy: 50 vs 49; reversible: 87 vs 81; thermodynamic(s): 117 vs 115). The differences appeared in passages, where Kaltura produced phonetically plausible but nonsensical substitutions while domain-prompted Whisper preserved meaning. Both systems failed on the rare proper noun "Helmholtz," rendered as "compulse" by each.

• Structural output differences matter downstream. Kaltura produces a continuous text wall with no segmentation or timestamps, whereas Whisper produces sentence-level segments with start and end timestamps, which the lecture analysis pipeline requires for question identification and confusion detection. Kaltura's JSON includes per-word confidence scores (garbled words: 0.32–0.67; correct domain terms: 0.88–1.00), but its plain-text export strips this metadata.

• Max-score merging outperformed weighted averaging for combining extractors. An earlier weighted-average approach (0.7× LLM + 0.3× regex) was abandoned because it suppressed matches found by only one extractor; a "fugacity coefficient" match scored by regex but missed by the LLM would receive a blended score of 6.3 (below threshold) instead of its true score of 21.

• Structured extraction scaled cheaply. The full corpus of 35 lectures was processed across all four analysis types in approximately 15 minutes on the RTX 4090 using Llama 3.1 8B via Ollama with a 16,384-token context window (lectures are approximately 13,000 tokens). The lecture 9 summary was produced in 5.6 seconds of inference time, and all 35 lectures produced schema-compliant summary output.

• Model quality issues persist in semantic analyses. Question identification mislabeled some instructor prompts (e.g., "What is entropy?") as student questions, and confusion detection tends toward over-sensitivity, sometimes flagging normal pedagogical rephrasing as confusion. The authors report that similar minor quality issues appear in all of the semantic tools.

• Transcript coverage was incomplete. Of 39 scheduled lecture sessions, 35 produced full transcripts (458–733 segments each), two sessions (010 and 025) were examinations with minimal proctor speech (2 and 5 segments), and two sessions (003 and 023) yielded no segments, likely due to cancellations or silent audio.

• Deployment is blocked by recording access, not by compute. The university's Kaltura-based lecture capture system provides no bulk download API, requiring manual, one-at-a-time retrieval, so end-of-semester batch processing is currently the most practical approach even though the technical pipeline can process a lecture in minutes.

Methodology in Plain English

The authors first ran all lecture recordings through a batch pipeline on a GPU workstation. Audio was extracted with ffmpeg at 16 kHz mono, then transcribed with the faster-whisper library using Whisper large-v3 in float16 precision, with Silero-VAD voice activity detection enabled to skip silence. A manually curated prompt of thermodynamic vocabulary biased the decoder toward correct terminology. Hallucination loops were addressed with three layers: a mild repetition penalty (repetition_penalty=1.1) and exact 3-gram blocking (no_repeat_ngram_size=3) at the decoder level, condition_on_previous_text=False to isolate context between decoding windows, and a deterministic post-processing pass that detects runs of three or more consecutive segments with identical text, keeps the first, removes the rest, and records loop metadata for auditing.

For the student tool, rather than embedding-based similarity search, they use the textbook's own back-of-book index as the retrieval artifact. A student query is sent to two extractors at once: a fast, deterministic regex extractor with a predefined list of multi-word technical phrases, and a local Llama 3.1 8B extractor that also expands the query with related concepts. Both feed the same hierarchical index search, which scores entries by term overlap weighted by term length, with bonuses for matches at the start of a topic name and, on the LLM path, for earlier pages. Results are merged by taking the maximum score for each unique index entry, filtered by threshold, sorted by page number, and the top five are assembled into a context string enriched with chapter and section information from the table-of-contents tree. That context plus the original query goes to Llama 3.1 8B at temperature 0.6, constrained by prompt to use only the provided context, never to invent chapter or section numbers, and to cite page references; a deterministic fallback handles the case where the model server is unavailable.

For the instructor tool, the same transcripts are processed with Llama 3.1 8B via Ollama in JSON mode over a 16,384-token context window through four analysis types: summaries, question identification, confusion detection, and anecdotes/analogies. Batch work runs on the RTX 4090 workstation while retrieval, vector search, and interactive querying run on a standard consumer laptop.

Why This Matters

Impact on research. The paper shifts attention from student-only AI tools to instructor-facing applications of the same infrastructure, and it provides a candid, quantitative account of deploying 7–8 billion parameter open-weight models for structured extraction over long transcripts. It also argues that a curated textbook index can be a better retrieval artifact than embedding-based similarity search over raw text, avoiding chunking decisions, an embedding model, a vector store, and similarity-threshold tuning.

Real-world applications:

• Course study aid. Students ask natural-language questions and receive grounded answers pointing to specific textbook chapters, sections, and pages, with examples such as the "Explain fugacity" query returning five merged index entries across Chapter 7, Section 7.4 and Chapter 9, Section 9.2.
• Instructor course analytics. Searching the transcript corpus lets an instructor trace when a concept was first discussed (e.g., "When did I first discuss fugacity?"), how much lecture time went to a topic, or whether an equation was connected back to earlier material.
• Question mining as continuous feedback. Logging student queries creates a passive feedback channel that surfaces recurring questions and misconceptions in real time, without relying solely on office hours or end-of-semester evaluations.
• Content reinforcement and course evolution. The system can identify material covered verbally in lecture but never appearing in written form, and comparing transcripts across semesters preserves institutional memory of how topics were treated.

Industry relevance. The paper's emphasis on fully local deployment with open-weight models addresses predictable costs (no per-query API fees), data privacy and control over student data, offline operation, and reproducibility independent of third-party service changes. The authors note that the GPU workstation represents a one-time hardware investment comparable to a single year of cloud API costs for equivalent throughput, and that models in the 7–13 B parameter range (e.g., Llama, Mistral, Qwen, Apertus) run comfortably on laptop-class hardware. The paper also flags a practical vendor lock-in problem, advocating for open API access to institutional lecture recordings.

Future Directions

• Enabling Stan to run interactive code examples, described by the authors as a planned extension.

• Systematizing domain-aware vocabulary prompts. Three strategies are proposed: automatically parsing the textbook table of contents and index to generate per-chapter or per-topic prompts that evolve with the course schedule; aligning prompts to the syllabus so each lecture uses a narrower, more targeted prompt; and iterative refinement in which instructor-identified misrecognitions are fed back into subsequent transcription runs, supported by storing all transcription parameters in the output JSON for reproducible re-transcription.

• Real-time, per-lecture analysis. The authors state that distributing a structured summary as a spaced retrieval cue before the next class is technically ready but deployment-blocked by the institutional recording workflow, and they advocate for open API access to lecture recordings. Direct USB microphone capture is offered as a workaround.

• Expanding coverage to cross-course and rare-term problems. The paper notes that even domain-prompted models struggle with terms uncommon in general speech corpora, such as "Helmholtz," suggesting further vocabulary expansion or fine-tuning, and envisions students supplementing Stan with notes from other courses (e.g., linear algebra or numerical methods) to run more complex cross-subject queries.

• Resolving copyright considerations for the textbook index. The authors recommend either seeking explicit publisher permission or replacing the published index with an instructor-generated topic list derived from the syllabus and lecture notes for production deployment.

Target Audience

Instructors and faculty in engineering and science disciplines who want to build or evaluate AI tools for their own courses; educational technology researchers and developers interested in RAG pipelines, local open-weight model deployment, and speech-to-text for technical content; and university administrators or IT staff considering the infrastructure, privacy, and recording-access implications of running such systems. Students in thermodynamics courses are end users of the tool rather than primary readers of the paper.

Authors’ abstract

Discussions of AI in education focus predominantly on student-facing tools -- chatbots, tutors, and problem generators -- while the potential for the same infrastructure to support instructors remains largely unexplored. We describe Stan, a suite of tools for an undergraduate chemical engineering thermodynamics course built on a data pipeline that we develop and deploy in dual roles: serving students and supporting instructors from a shared foundation of lecture transcripts and a structured textbook index. On the student side, a retrieval-augmented generation (RAG) pipeline answers natural-language queries by extracting technical terms, matching them against the textbook index, and synthesizing grounded responses with specific chapter and page references. On the instructor side, the same transcript corpus is processed through structured analysis pipelines that produce per-lecture summaries, identify student questions and moments of confusion, and catalog the anecdotes and analogies used to motivate difficult material -- providing a searchable, semester-scale record of teaching that supports course reflection, reminders, and improvement. All components, including speech-to-text transcription, structured content extraction, and interactive query answering, run entirely on locally controlled hardware using open-weight models (Whisper large-v3, Llama~3.1 8B) with no dependence on cloud APIs, ensuring predictable costs, full data privacy, and reproducibility independent of third-party services. We describe the design, implementation, and practical failure modes encountered when deploying 7--8 billion parameter models for structured extraction over long lecture transcripts, including context truncation, bimodal output distributions, and schema drift, along with the mitigations that resolved them.