A Multimodal Dataset of Student Oral Presentations with Sensors and Evaluation Data

AI summary

Overview

Research area: Human-Computer Interaction / Multimodal Learning Analytics (MMLA) in education, specifically the sensing and evaluation of student oral presentations in real classrooms.

Technical level: Intermediate. The paper is a dataset descriptor rather than an algorithmic contribution, so the motivation and rubric design are broadly accessible; the sensor specifications, synchronization scheme, and preprocessing pipeline assume some familiarity with multimodal data collection.

Scope: SOPHIAS is a 12-hour, 385 GB, controlled-access multimodal dataset of 50 real classroom oral presentations (46 individual undergraduate, 4 master's group) delivered by 65 students, combining eight sensor streams with three-source rubric evaluations and timestamped contextual annotations.

What This Paper Is About

Most tools that automatically analyze oral presentation skills are tested on very small samples in laboratory settings, which limits how well they transfer to real classrooms. The authors address this gap by collecting and releasing SOPHIAS, a dataset of genuine face-to-face student presentations recorded under ordinary teaching conditions, in which video, audio, eye-tracking, smartwatch physiology, slide interactions, and rubric scores are all synchronized on a common timeline. The goal is to give researchers a realistic benchmark for studying how behavioral and physiological signals relate to presentation quality and for building automated feedback systems.

Key Contributions

1. An ecologically valid multimodal corpus for oral presentations. SOPHIAS captures 50 presentations from 65 undergraduate and master's students at the Universidad Autónoma de Madrid in a regular classroom, preserving spontaneous behavior rather than scripted lab performances. The dataset comprises 385 GB and 12 hours of presenter recordings.

2. Eight synchronized sensor streams across three platforms plus a wearable acquisition tool. Data were gathered using edBB (video, webcam audio, screen recording), Microsoft Teams (ambient audio), the AICoFe system (assessments, comments, keystroke logs), and Watch-DMLT for Fitbit Sense smartwatches. Modalities include HD webcam video, ambient and webcam audio, Tobii Pro Glasses 3 eye-tracking, smartwatch physiological and inertial signals, clicker/keyboard/mouse events, PDF slides, and derived face/head-pose features.

3. Three complementary sources of evaluation within the same sessions. Every presentation is scored with a standardized analytic rubric, developed collaboratively by five UAM teachers, by teachers, peers, and the presenter's own self-assessment, with free-text comments and typed-keystroke timestamps. Rubric items are rated on a 5-point Likert scale (1 = Very Poor, 5 = Excellent).

4. Processed features, schemas, and documentation for reproducible reuse. The release includes RetinaFace + WHENet-derived face bounding boxes, five facial landmarks, and head pose angles (yaw, pitch, roll) at 20 Hz, plus machine-readable JSON Schemas and a dataset_catalog.json in a public GitHub repository to support automated validation and parsing.

Main Findings

• Presentation durations and structure: The 46 individual undergraduate presentations had a mean duration of 9 min 48 s (SD = 33 s; range 8 min 08 s–10 min 38 s), followed by a mean Q&A of 6 min 28 s (SD = 3 min 02 s; range 2 min 13 s–14 min 41 s). The 4 master's group presentations had a mean duration of 14 min 11 s (SD = 1 min 40 s; range 12 min 54 s–16 min 38 s), followed by a mean Q&A of 8 min 22 s (SD = 1 min 21 s; range 6 min 45 s–9 min 29 s).

• Participant composition: 46 undergraduate students (40 male, 6 female) in the Telecommunication Technology and Service Engineering degree with a mean age of 21.6 years (SD = 0.84; range 21–25), and 19 master's students (14 male, 5 female) from the Master in Data Science with a mean age of 23.6 years (SD = 1.26; range 22–27). Master's students formed groups of 4 or 5, which is why the student count exceeds the presentation count.

• Session timing: Undergraduate presentations took place between March and May 2025; all master's presentations took place on the same day in May 2025.

• Sensor specifications: Two Logitech C920 PRO HD webcams (presenter view and evaluators' view) recorded at 20 Hz and 1920×1080; a Poly Sync 40 omnidirectional microphone captured ambient audio at 16 kHz while webcam-integrated microphones recorded at 8 kHz; Tobii Pro Glasses 3 provided gaze at 50 Hz, first-person video at 25 Hz and 1920×1080, and IMU data at 120 Hz; Fitbit Sense smartwatches recorded heart rate (PPG), accelerometer, gyroscope, and orientation at 1 Hz.

• Synchronization approach: Timestamped files carry both local time (Europe/Madrid) and UNIX time in milliseconds since the UNIX epoch, with UNIX time used as the common reference across platforms and sensors.

• Classroom geometry: The room measured approximately 10.40 m wide and 12.90 m long. The presenter webcam sat roughly at the centre of the classroom width, about 5 m from the projection screen, with the presenter standing approximately 4.00–4.10 m from the webcam.

• Rubric structure: Undergraduate presentations were rated on individual-level criteria across three dimensions—Content, Design, and Presenter Performance. The master's rubric separated group-level criteria (content, design, coordination) from individual-level criteria (voice and tone, body language, eye contact, autonomy, handling of questions, mastery of the topic).

• Privacy handling: Facial blurring was deliberately not applied to the released MP4 recordings because facial expressions, gaze direction, head pose, and body posture are core research signals. Instead, institutional identifiers were replaced with random internal identifiers generated through the AICoFe system, and participant names detected in audio were replaced with silence at the original timings while preserving recording duration.

• Preprocessing details: Heart rate readings were smoothed with a 15-second moving average; accelerometer data were filtered with a fourth-order Butterworth low-pass filter at 0.2 Hz; gyroscope data used an equivalent Butterworth topology and cutoff frequency. Both raw and filtered smartwatch signals are provided. Webcam and Teams audio were aligned using FFT-based cross-correlation, and name censorship used faster-whisper transcription with approximate string matching (SequenceMatcher).

• Comparative positioning: Table 1 contrasts SOPHIAS with prior resources including the Math Data Corpus (18 students; 12 sessions), the Public Speaking / Oral Presentation Quality Corpus (17 speakers), mEBAL and mEBAL2 (38 students in mEBAL; 180 students in mEBAL2), IMPROVE (120 learners), and the Lecture Presentations Multimodal Dataset (10 lecturers; 180+ hours of video). SOPHIAS is positioned as the entry focused on real face-to-face oral presentations that combines synchronized multimodal sensing, slides, interaction logs, contextual annotations, and three-source assessments in a single classroom-based dataset.

• Ethics and consent: Collection followed the Declaration of Helsinki and was approved by the Ethics Committee of the Universidad Autónoma de Madrid (Approval No. CEI-155-3850), within teaching innovation project EPS_021.24_INN. Participation, partial participation, or refusal had no effect on course grades.

• Access model: The dataset is hosted under controlled access in the Science Data Bank (DOI: https://doi.org/10.57760/sciencedb.33655), with code, schemas, catalog, and documentation on GitHub at https://github.com/dataGHIA/SOPHIAS. Access requires approval of a Data Usage Agreement; redistribution of original data, extracted features, encrypted copies, or derived data is prohibited.

Methodology in Plain English

The team set up a multimodal recording infrastructure in an ordinary university classroom rather than a lab. Before any recording, students attended two preparatory sessions: one on oral presentation skills, and one explaining the peer- and self-assessment methodology, the wearable sensors, and the rubric and its level descriptors, so that evaluators would apply the criteria consistently. Students then gave their presentations as a normal part of their courses—undergraduates individually on randomly assigned engineering topics, master's students in groups of four or five presenting self-selected Data Science topics they had first written up as academic papers.

During each session, a fixed cast of roles was recorded: the presenter, the teacher evaluator, two peer evaluators, and a student acting as an external observer who wore the eye-tracking glasses after calibration. A research assistant annotated contextual events in real time—nervous movements, reading from slides or notes, eye contact with the audience, filler words, and the start and end of each question and answer. Cameras, microphones, smartwatches, and interaction logs ran simultaneously, and all streams were anchored to UNIX timestamps so they could be aligned later.

Afterwards, every presentation was scored with the same analytic rubric through the AICoFe platform by teachers, peers, and the presenter themselves; students watched their own recording for the self-assessment and then received personalized feedback generated by the GePeTo AI module. Finally, the authors processed the video with a RetinaFace and WHENet pipeline to extract face boxes, facial landmarks, and head pose, synchronized the two audio tracks by cross-correlation, filtered the smartwatch signals, censored spoken names, replaced institutional identifiers with random internal IDs, and packaged everything into 50 presentation folders with accompanying schemas and a dataset catalog.

Why This Matters

Impact on research. The paper argues that existing MMLA tools for oral presentations are typically validated with very small samples in controlled laboratory environments, which limits their external validity. SOPHIAS provides a larger, classroom-grounded resource with synchronized modalities plus three evaluation perspectives, enabling research on links between physiological and behavioral signals and presentation performance, on the reliability of peer assessment, and on automated feedback. It also carries the timestamped contextual annotations needed as labels for supervised learning and for validating automatic detectors built from video, audio, heart rate, or gaze.

Real-world applications.

• Automated feedback systems that give students targeted, evidence-based coaching on delivery, engagement, and slide use.
• Peer- and self-assessment training, using the multi-source rubric scores to study calibration and reliability across evaluators.
• Public-speaking and communication-skills instruction in higher education, supported by objective behavioral and physiological indicators.
• Early detection of presentation anxiety or stress from heart rate and motion signals, informing support for struggling speakers.
• AI-assisted behavioral and nonclinical assessment research that relies on gaze, facial expression, attention, and verbal patterns.

Industry relevance. The dataset's controlled-access terms explicitly permit academic and legitimate commercial research and development, so companies building communication-coaching platforms, meeting-analytics tools, or interview and presentation training products could use it as a benchmark. It also speaks to broader interest in wearable and camera-based sensing for human behavior analysis.

Future Directions

• Developing and benchmarking automated feedback and Multimodal Learning Analytics models on SOPHIAS, particularly for detecting events such as nervous movements, reading behavior, and eye contact from sensor streams.
• Investigating how physiological signals (heart rate, motion) and behavioral cues relate to rubric scores and to the discrepancies between teacher, peer, and self-assessments.
• Assessing whether the dataset's scale and ecological validity improve generalization compared with the smaller, lab-based corpora listed in Table 1.
• Analyzing the post-test learning gains the protocol collected after all presentations, and whether the MOSAIC-F framework and GePeTo feedback module measurably improve presentation skills.
• Broader open question: whether models trained on presentations from one institution, language, discipline, and classroom layout transfer to other educational settings, and how data-access constraints affect reproducibility.

Target Audience

Researchers and practitioners in Multimodal Learning Analytics, educational data mining, and human-computer interaction who need realistic multimodal data on oral presentations. It is also relevant to instructors and teaching-innovation teams interested in rubric design and peer/self-assessment, to developers building automated presentation-feedback or public-speaking coaching tools, and to affective-computing or sensor-based behavior researchers who require synchronized video, audio, gaze, and physiological signals with human-annotated labels.

Authors’ abstract

Oral presentation skills are a critical component of higher education, yet comprehensive datasets capturing real-world student performance across multiple modalities remain scarce. To address this gap, we present SOPHIAS (Student Oral Presentation monitoring for Holistic Insights & Analytics using Sensors), a 12-hour multimodal dataset containing recordings of 50 oral presentations delivered by 65 undergraduate and master's students at the Universidad Autonoma de Madrid, comprising 46 individual presentations with a mean presentation duration of 9 min 48 s (SD = 33 s) followed by a mean Q&A duration of 6 min 28 s (SD = 3 min 02 s), and 4 group presentations with a mean presentation duration of 14 min 11 s (SD = 1 min 40 s) followed by a mean Q&A duration of 8 min 22 s (SD = 1 min 21 s). SOPHIAS integrates eight timestamped sensor streams from high-definition webcams, ambient and webcam audio, eye-tracking glasses, smartwatch physiological sensors, and clicker, keyboard and mouse interactions. In addition, the dataset includes slides and rubric-based evaluations from teachers, peers, and self-assessments, along with timestamped contextual annotations. The dataset captures presentations conducted in real classroom settings, preserving authentic student behaviors, interactions, and physiological responses. SOPHIAS enables the exploration of relationships between multimodal behavioral and physiological signals and presentation performance, supports the study of peer-assessment and provides a benchmark for developing automated feedback and Multimodal Learning Analytics tools. The dataset is available for research, including both academic and legitimate commercial research and development, under controlled access through Science Data Bank, subject to approval of a Data Usage Agreement (DUA), with code provided through GitHub.