ViBED-Net: Video Based Engagement Detection Network Using Face-Aware and Scene-Aware Spatiotemporal Cues

AI summary

Overview

• Research area: Video-based affective computing and educational technology, specifically automatic detection of student engagement from video.
• Technical level: Intermediate. The paper assumes familiarity with convolutional neural networks, recurrent networks (LSTM), and Transformer self-attention, but explains each component in enough detail to follow without prior work in engagement detection.
• One-sentence scope: ViBED-Net is a dual-stream deep learning framework that combines face-cropped and full-frame video streams, EfficientNetV2 spatial features, and LSTM or Transformer temporal modeling to classify student engagement levels on the DAiSEE dataset.

What This Paper Is About

Online learning platforms need to know whether students are actually paying attention, but detecting engagement automatically from video is hard because engagement shows up both in subtle facial expressions and in broader body posture and scene context. The authors build a two-stream model that looks at the whole video frame and at a cropped face region at the same time, then models how those features change across a clip. The goal is to beat existing accuracy on the DAiSEE benchmark while handling the dataset's heavy class imbalance.

Key Contributions

1. A dual-stream architecture (ViBED-Net) that processes full video frames for scene/context cues and OpenCV-cropped facial regions for fine-grained expression cues in parallel, rather than relying on only one of the two.
2. A comparison of two temporal modeling strategies on identical spatial features: LSTM networks (the default) and a Transformer encoder with positional encoding, multi-head self-attention, and 2 encoder layers.
3. Targeted data augmentation for underrepresented engagement classes, using salt-and-pepper noise, random translations, brightness and contrast adjustments, Gaussian blur, horizontal flips, and elastic transformations.
4. A new state-of-the-art accuracy on DAiSEE of 73.43% with the LSTM variant, reported against eight previously published methods, plus a public source code release at https://github.com/prateek-gothwal/ViBED-Net.

Main Findings

• LSTM beats Transformer: ViBED-Net with LSTM reaches 73.43% overall accuracy on the DAiSEE test set, while the Transformer-based variant reaches only 62.39%.
• Best prior baseline outperformed: The strongest comparison method, a "General Model for Learner Engagement," reported 68.57%. Other listed baselines include EfficientNetB7 + LSTM (67.48%), BiusFPN + ICCSA (68.16%), Affect-driven Ordinal Engagement (67.4%), ResNet + TCN Hybrid (63.9%), VisioPhysioENet (63.09%), DFSTN (58.84%), CNN-RNN Baseline (53.9%), and I3D (52.35%).
• Minority class behavior, LSTM: For "Very Low" engagement the LSTM achieves precision of 1.00 but recall of only 0.25, meaning it misses 75% of actual "Very Low" instances. F1-scores are 0.40 (Very Low), 0.80 (Low), 0.72 (High), and 0.74 (Very High).
• Transformer collapses on minority classes: The Transformer scores 0.00 precision, recall, and F1 on "Very Low," and only 0.13 recall with a 0.22 F1 on "Low." Its best class is "High" with a 0.67 F1.
• Errors concentrate between adjacent levels: The confusion matrix shows most misclassifications occur between neighboring engagement levels (for example "Low" versus "High"), which the authors attribute to the subjective nature of engagement annotation.
• Class imbalance remains unsolved: Despite augmentation, the authors state the model struggles to identify all instances of the underrepresented "Very Low" class.

Methodology in Plain English

The team treats a 10-second video clip as a sequence of frames rather than a single image. For every clip they uniformly sample 60 frames. Each frame goes through two parallel EfficientNetV2 networks: one that sees the entire frame (capturing posture, body language, background) and one that sees only a face region cropped out with OpenCV. The classification head is removed from EfficientNetV2, and features are taken from the penultimate layer, giving a vector of size 1028 per frame, or a (60, 1028) sequence per stream. These features are precomputed and saved as pickle files so training does not repeat the expensive extraction step.

Each of the two feature sequences is then fed into its own temporal model. In the default LSTM setup, each LSTM has input size 1028, hidden size 1024, 1 layer, and dropout of 0.1; the final hidden states from the face stream and the frame stream are concatenated. In the alternative Transformer setup, positional encodings are added, followed by 2 encoder layers with 8 attention heads, hidden dimension 1024, and feed-forward layers of size 2048, with layer normalization and residual connections. The concatenated representation (2048 dimensions from the two 1024-dimension streams) passes through an MLP that reduces it to 512 with ReLU and dropout of 0.3, then to the number of engagement classes (4 in DAiSEE).

Training uses cross-entropy loss and the AdamW optimizer with a learning rate of 0.001, mini-batch size 32, and 40 epochs, with validation accuracy monitored per epoch. Evaluation reports accuracy, precision, recall, and F1 as macro-averages to give equal weight per class.

Why This Matters

Research impact. The paper shows that for engagement detection, how you fuse spatial information (face plus scene) and which temporal model you choose both matter measurably — the LSTM and Transformer variants differ by more than ten accuracy points on identical features. It also provides an explicit, quantified example of the class-imbalance problem in affective datasets, where a model can be perfectly precise yet barely detect a minority class.

Real-world applications:

• Online learning analytics, where instructors or platforms could flag disengaged students for intervention or personalize instruction.
• Content creation and entertainment, where creators could identify the most captivating segments of films or games.
• Marketing analytics and user experience research, where attention and affective responses to content inform design decisions.
• Gaming and human-computer interaction, where a system adapts to a user's attention state in the moment.

Industry relevance. The modular design and precomputed feature pipelines make the approach practical to adapt across domains, and the public code release lowers the barrier to reuse. The authors note, however, that reliance on precomputed features may not suit fully real-time, end-to-end deployment, and that the model assumes a relatively consistent, front-facing camera view.

Future Directions

• Add more modalities, such as eye-gaze tracking, to get a more direct and holistic measure of visual attention alongside facial and scene cues.
• Make the model view-invariant, so it works with different camera placements and viewing angles through more diverse training data or view-invariant feature learning.
• Apply self-supervised learning on large unlabeled video datasets to learn more generalizable representations of human behavior, then fine-tune for engagement detection to improve cross-dataset robustness.
• Move toward real-time, end-to-end deployment rather than depending on cached EfficientNetV2 features, and continue addressing the low recall on the "Very Low" engagement class.

Target Audience

This paper is most useful to graduate students and researchers working on affective computing, video understanding, or educational data mining, and to practitioners building engagement or attention analytics into learning platforms, media, or user research tools. It also suits engineers who want a clear, reproducible template for combining a CNN backbone with either recurrent or Transformer temporal modeling, since the architecture choices, hyperparameters, and code are all reported.

Authors’ abstract

Engagement detection in online learning environments is vital for improving student outcomes and personalizing instruction. We present ViBED-Net (Video-Based Engagement Detection Network), a novel deep learning framework designed to assess student engagement from video data using a dual-stream architecture. ViBED-Net captures both facial expressions and full-scene context by processing facial crops and entire video frames through EfficientNetV2 for spatial feature extraction. These features are then analyzed over time using two temporal modeling strategies: Long Short-Term Memory (LSTM) networks and Transformer encoders. Our model is evaluated on the DAiSEE dataset, a large-scale benchmark for affective state recognition in e-learning. To enhance performance on underrepresented engagement classes, we apply targeted data augmentation techniques. Among the tested variants, ViBED-Net with LSTM achieves 73.43\% accuracy, outperforming existing state-of-the-art approaches. ViBED-Net demonstrates that combining face-aware and scene-aware spatiotemporal cues significantly improves engagement detection accuracy. Its modular design allows flexibility for application across education, user experience research, and content personalization. This work advances video-based affective computing by offering a scalable, high-performing solution for real-world engagement analysis. The source code for this project is available on https://github.com/prateek-gothwal/ViBED-Net .