Context Matters: Peer-Aware Student Behavioral Engagement Measurement via VLM Action Parsing and LLM Sequence Classification

AI summary

Overview

Research area: Computer vision for education — video-based student behavioral engagement measurement, combining vision-language models (VLMs) for action recognition with large language models (LLMs) for sequence classification.

Technical level: Intermediate. Readers should be comfortable with transfer learning, few-shot adaptation, CLIP-style vision-language models, temporal action segmentation metrics, and prompt-based LLM classification.

Scope: The paper proposes and evaluates a three-stage, peer-aware pipeline that turns a 2-minute classroom video of a student into an ordered action sequence, then uses an LLM to label that sequence as engaged or disengaged in light of what peers are doing.

What This Paper Is About

Automatically judging whether a student is engaged in class usually requires large amounts of annotated video, and most existing work ignores what the rest of the classroom is doing. This paper argues that the same observable behavior — typing on a laptop, turning to talk to a peer — can indicate either engagement or disengagement depending on the surrounding classroom activity (the ICAP Passive–Active–Constructive–Interactive framework is cited for this point).

The goal is a training-light framework that recognizes fine-grained student actions with only a few labeled examples per class, converts them into an ordered sequence over a 2-minute window, and lets an LLM decide engagement while also seeing the majority actions of the student's peers.

Key Contributions

1. A novel three-stage framework for student behavioral engagement: few-shot VLM action recognition, sliding-window action parsing into a temporal sequence, and LLM-based engagement classification. Only the VLM stage is fine-tuned; the parsing and engagement stages are training-free (sliding window plus zero-shot LLM prompting).
2. A peer-aware (context-aware) engagement classifier that feeds both the target student's action sequence and the classroom context — represented as the majority of peers' actions during the interval — into the LLM.
3. A new student behavior dataset with two independently annotated components: (a) 208 trimmed training clips (16 per action) plus 46 test clips across 13 classroom action categories in a controlled environment, and (b) untrimmed lecture video from three half-hour lectures with 11 students, giving 455 two-minute clips labeled engaged/disengaged (400 engaged, 55 disengaged) with dense temporal action labels for each segment.
4. A direct comparison against a prior histogram-based engagement method, plus ablations isolating the contribution of classroom context and of sequence-versus-histogram input representation.

Main Findings

• Zero-shot VLM action recognition is weak in this domain: ViFi-CLIP reached 52.1% top-1 accuracy on the 46-clip student action test set, which the authors attribute to its fine-tuning on general web video (Kinetics-400) failing to capture fine-grained distinctions such as reading versus writing.
• Few-shot adaptation works well: In Table 1, ViFi-CLIP scored 47.7% (K=2), 81.8% (K=4), 93.2% (K=8), and 97.9% (K=16); XCLIP scored 43.2%, 79.6%, 88.6%, and 97.9%; TC-CLIP scored 52.2%, 52.2%, 80.4%, and 93.5%. ViFi-CLIP gains over XCLIP were 4.5%, 2.2%, and 4.6% at K=2, K=4, and K=8 respectively.
• Gemini-2.5-Pro parses actions better than the proposed VLM parser, which in turn beats Gemini-2.5-Flash: On temporal action segmentation (Table 2), accuracy/edit/F1@10/F1@25/F1@50 were 57.2/37.4/39.0/34.5/25.3 for Gemini-2.5-Flash, 69.8/51.8/58.2/55.2/45.2 for Gemini-2.5-Pro, and 67.0/45.7/48.3/43.9/31.4 for the VLM-based approach.
• Best engagement classification comes from manual action labels: With human-annotated actions (Table 3), Gemma-2-9B achieved a weighted-average F1 of 0.92, Llama-3-8B 0.91, and GPT-3.5-turbo 0.89. The paper's stated conclusion reports F1 scores of 92%, 86%, and 88% using manual, Gemini-based, and modified VLM-based temporal action segmentation respectively.
• Automated parsing costs some accuracy, but the proposed parser holds up: With Gemma-2-9B (Table 4), Gemini-2.5-Pro-based parsing gave a weighted-average F1 of 0.86 and VLM-based parsing 0.88.
• Context-aware classification beats context-free: Removing peer context (Table 7) dropped the disengaged-class F1 from 0.52 to 0.51 (a 1% difference as reported) and pushed disengaged precision below 50% (0.44).
• Sequences beat histograms: Comparing input representations (Table 8), the sequence-based method reached a weighted-average F1 of 0.88 versus 0.86 for the histogram; on the disengaged class the gap was 0.52 versus 0.39.
• The framework outperforms the prior histogram-based method: Against the approach of Abdelkawy et al. [2] evaluated with Leave-One-Out Cross-Validation (Table 5), the proposed framework scored 0.88 weighted-average F1 versus 0.85 — reported as a 3% improvement — and 0.52 versus 0.47 on the disengaged class.
• It generalizes to a different class: On an additional 30-minute lecture from a separate class with eight different students (113 two-minute segments: 73 engaged, 40 disengaged), VLM-based parsing achieved 0.83 weighted-average F1, compared with 0.72 for the Gemini-2.5-Pro-based variant (a margin reported as 9%) and 0.88 with manual annotation (Table 6).
• Disengaged detection remains the hard class: The authors attribute moderate disengaged-class performance to inherent ambiguity — students doodling on tablets were labeled disengaged but the action set does not distinguish writing from drawing, daydreaming students were sometimes annotated as listening, and students at the classroom periphery may appear to look forward without attending.

Methodology in Plain English

The pipeline has three stages.

Stage 1 — Learn student actions from few examples. The team fine-tuned ViFi-CLIP, a video version of CLIP with a ViT-B/16 backbone, using only 16 video clips per action category (K=16), with a batch size of 32, learning rate of 2×10⁻⁶, 50 epochs, AdamW with weight decay 0.001, and cosine annealing. Each clip is represented by 16 uniformly sampled frames resized to 224×224. Training maximizes cosine similarity between averaged frame embeddings and prompt-based text embeddings of the action name, and adds an entropy-regularization term to the cross-entropy loss so the model does not become overconfident and overfit.

Stage 2 — Turn a 2-minute video into an action sequence. Each student's 2-minute tubelet is cut into non-overlapping segments with a sliding temporal window (window size and stride both 3 seconds, i.e., 45 frames), producing 40 segments per 2-minute video. The fine-tuned VLM labels each segment, and consecutive identical predictions are merged so the sequence preserves both action identity and duration.

Stage 3 — Classify engagement with an LLM. The ordered sequence, with timestamps for each action, is placed in a prompt along with the classroom context (the majority actions of peers over the same interval). A zero-shot LLM (Gemma-2-9B is used for most experiments, with Llama-3-8B and GPT-3.5-turbo also tested) outputs engaged or disengaged. Temperature is set to 0.1 for deterministic output.

Evaluation uses task-specific metrics: top-1 accuracy for action recognition; Mean over Frames (MoF), Edit Score, and F1@τ (τ ∈ {10, 25, 50}) for action parsing; and precision, recall, and F1 for engagement classification, chosen over plain accuracy because of the engaged/disengaged class imbalance. Videos were captured by a wall-mounted 4K camera at 15 fps in classrooms of 6–11 students. The study was approved under IRB # 19.0513 and funded by NSF Award Number 2337154.

Why This Matters

Impact on research: The paper shows that engagement is not a property of a single student's behavior in isolation — it depends on peer context — and that off-the-shelf LLMs can perform this contextual reasoning zero-shot. It also reframes student behavior as an ordered sequence rather than an aggregated histogram, and releases a dataset with dual expert annotations (engagement labels plus dense temporal action labels) to address the scarcity of public benchmarks in this area.

Real-world applications:

• Automated classroom observation as a scalable alternative to manual protocols, which the paper notes are labor-intensive and subject to observer bias.
• Quantitative feedback for instructors, letting them adjust lectures and classroom activities to re-engage disengaged students.
• Early identification of off-task behavior patterns (for example, frequent short phone checks versus one long interruption) that a duration-only summary would miss.
• Cross-course deployment: the framework was tested on a lecture from a different class with different students and instructors and retained reasonable performance.

Industry relevance: Relevant to educational technology and classroom-analytics vendors, camera-based sensing platforms for schools and universities, and teacher professional-development tools. The paper's emphasis on few-shot tuning and zero-shot LLM prompting also matters commercially because it reduces the annotation cost that has kept most prior work locked to proprietary datasets.

Future Directions

• Improve fine-grained action parsing. The action dictionary does not separate writing from drawing or analogous visually similar behaviors, which the authors identify as a source of labeling ambiguity penalizing the disengaged class.
• Close the gap to manual annotation. Even the best automated parsing yields a weighted-average F1 of 0.88 versus 0.92 with manual action labels, so better segmentation or use of Gemini-2.5-Pro-level models could raise the ceiling.
• Scale the dataset. The evaluation covers three half-hour lectures plus one additional 30-minute lecture from a second class; broader multi-course, multi-institution validation is an open question the authors raise themselves.
• Handle difficult visual conditions. Inter-student occlusion and cluttered classroom environments are explicitly named as challenges, as is the case of students at the classroom periphery who appear to look forward without attending.

Target Audience

Researchers and practitioners in computer vision, multimodal learning, and learning analytics who work on classroom behavior analysis or human action understanding; education researchers and instructional designers interested in the ICAP framework and in automating behavioral engagement measurement; and developers building educational technology systems that need privacy-conscious, low-annotation pipelines for interpreting student behavior in video.

Authors’ abstract

Understanding student behavior in the classroom is essential to improve both pedagogical quality and student engagement. Existing methods for predicting student engagement typically require substantial annotated data to model the diversity of student behaviors, yet privacy concerns often restrict researchers to their own proprietary datasets. Moreover, the classroom context, represented in peers' actions, is ignored. To address the aforementioned limitation, we propose a novel three-stage framework for video-based student engagement measurement. First, we explore the few-shot adaptation of the vision-language model for student action recognition, which is fine-tuned to distinguish among action categories with a few training samples. Second, to handle continuous and unpredictable student actions, we utilize the sliding temporal window technique to divide each student's 2-minute-long video into non-overlapping segments. Each segment is assigned an action category via the fine-tuned VLM model, generating a sequence of action predictions. Finally, we leverage the large language model to classify this entire sequence of actions, together with the classroom context, as belonging to an engaged or disengaged student. The experimental results demonstrate the effectiveness of the proposed approach in identifying student engagement. The source code will be available at https://github.com/ahmed-nady/context_aware_student_engagement.