EgoEMS: A High-Fidelity Multimodal Egocentric Dataset for Cognitive Assistance in Emergency Medical Services

AI summary

Overview

Research area: Egocentric multimodal machine learning for healthcare, specifically human activity recognition and cognitive assistance in Emergency Medical Services (EMS).

Technical level: Intermediate — readers should be comfortable with activity recognition terminology (classification, segmentation, keysteps), multimodal sensor fusion, and standard evaluation metrics (top-1 accuracy, RMSE, F1).

Scope: This paper introduces EgoEMS, a 20-hour multimodal egocentric dataset of simulated EMS procedures with 233 trials by 62 participants, plus benchmark results for keystep classification, keystep segmentation, and CPR quality estimation.

What This Paper Is About

EMS responders operate under severe cognitive load in high-stakes emergencies, and AI cognitive assistants could help track protocol compliance and provide real-time feedback. However, no existing dataset captures the procedural complexity, multiperson coordination, and multimodal sensing required to build such systems — prior medical egocentric datasets are small, single-view, limited to a few procedures, or lack ground-truth annotations. EgoEMS fills this gap by providing a large, synchronized, richly annotated egocentric dataset of realistic simulated EMS scenarios, along with baseline benchmarks that reveal where current models succeed and fall short.

Key Contributions

1. First synchronized multimodal egocentric dataset of multiperson EMS procedures. EgoEMS captures 20 hours of data across 233 trials from 62 participants (46 EMS professionals, 16 public volunteers), including synchronized egocentric video, scene audio, smartwatch IMU, and ground-truth CPR quality metrics from a manikin-mounted Time-of-Flight sensor.

2. A hierarchical EMS taxonomy aligned with national standards. Working with EMS experts and drawing on the NEMSIS database and NREMT psychomotor guidelines, the authors defined 3 high-frequency protocols (Cardiac Arrest, Cardiac Suspected, Stroke), 9 critical interventions, and 67 fine-grained keysteps, plus a set of key medical objects/tools.

3. Rich multi-level annotations. The dataset includes manual keystep labels (2,694 instances), timestamped speaker-diarized transcripts (140), semi-automatic bounding boxes (13.7k) and segmentation masks (12k), and CPR compression rate/depth ground truth (169), with a de-identification pipeline for privacy compliance.

4. A suite of benchmarks for real-time inference. Three tasks — keystep classification, online keystep segmentation, and CPR quality estimation — are evaluated with supervised transformers, CNN baselines, few-shot cross-domain models, and zero-shot LLMs (Qwen-2.5, VideoLLaMA-3.3, Whisper+GPT-4o).

Main Findings

• Keystep classification tops out at ~62% top-1 accuracy. A supervised transformer with ResNet50 video features reaches 62.3%; fusing smartwatch IMU with video yields 62.2% — early fusion gave no measurable gain, suggesting more sophisticated fusion is needed. Zero-shot Qwen-2.5 reaches 38.3%, indicating LLMs can partially recognize EMS actions without task-specific training.

• Multimodal fusion matters more for segmentation. In online 5-second-window keystep segmentation, combining egocentric video with IMU improves accuracy by 6 percentage points, reaching 61% — the short temporal windows make complementary modalities more valuable. Zero-shot Qwen-2.5 still achieves 55.5%, while audio-only Whisper+GPT-4o trails at 38%, largely because responders do not verbalize every action during critical interventions.

• CPR quality estimation benefits from combined sensing. Fusing video and IMU produces the best overall CPR feedback (F1 = 0.52 for compression rate, 0.83 for compression depth). IMU alone gives the lowest RMSE for compression rate; fusion gives the lowest RMSE for depth.

• Skill level visibly affects CPR consistency. Ground-truth rate distributions show EMS professionals maintain steady compression rates, while public participants exhibit much greater variability — a signal that could inform training and feedback systems.

• Semi-automatic annotation saves substantial effort. Object bounding-box and segmentation annotations, generated by a fine-tuned detector plus zero-shot segmentation with manual verification on 10%, saved over 60 hours of annotation time at a slight precision cost.

Methodology in Plain English

The team first analyzed the national NEMSIS database to identify which EMS protocols occur most often and matter most for survival. With EMS experts, they decomposed those protocols into 9 interventions and 67 discrete keysteps matching NREMT training standards.

They then ran 233 IRB-approved simulated emergency trials. Some scenarios used human actors as patients for high-fidelity realism (ambulance transfers, interfering bystanders, varied medical histories); cardiac arrest trials used manikins so CPR could be performed safely and measured. Each trial involved 2–3 responders, with 46 certified EMS personnel and 16 laypeople participating.

Hardware was kept deliberately cheap and replicable: a chest-mounted GoPro HERO for the egocentric view and audio, a Samsung Galaxy Watch 5 for hand-motion IMU, and a custom VL6180X Time-of-Flight sensor embedded in the manikin to record chest compression depth. All streams were aligned via Unix timestamps.

Annotations were produced through a mix of expert manual labeling (keysteps, verification), LLM-assisted pipelines (speaker-diarized transcripts), and semi-automatic detection-plus-segmentation (medical tools), each followed by human verification. Privacy protection involved face blurring, manual removal of names, license plates, and ID cards, all under IRB oversight.

Finally, they benchmarked three tasks — keystep classification, online keystep segmentation, and CPR quality estimation — using a diverse set of models spanning supervised transformers, CNNs, few-shot methods, and zero-shot LLMs, to establish baselines and expose current limitations.

Why This Matters

Impact on research: EgoEMS provides the first dataset where researchers can study end-to-end EMS workflows with synchronized multimodal sensing, multiperson dynamics, and clinically grounded ground truth. It shifts the egocentric activity recognition community's focus from daily-life chores toward safety-critical procedural domains, and it supplies a benchmark suite that lets researchers directly compare supervised, few-shot, and zero-shot approaches on the same realistic task.

Real-world applications:

• Real-time cognitive assistants worn by responders that track which keystep was performed, flag missed steps, and prompt the next action during time-critical interventions like CPR or stroke assessment.
• Automated CPR coaching that uses video and IMU to estimate compression rate and depth, delivering continuous feedback to improve adherence to 100–120 compressions/min and ≥50 mm depth guidelines.
• Training and skills assessment for EMTs, paramedics, and laypeople — the dataset captures a wide range of skill levels, enabling objective evaluation and personalized feedback during simulation training.
• Clinical decision support in stroke and cardiac emergencies, where adherence to protocol timing windows directly affects neurological and survival outcomes.

Industry relevance: The low-cost, open-source capture system (off-the-shelf GoPro, smartwatch, custom ToF sensor) lowers the barrier for medical device companies, EMS software vendors, and AR/VR platform developers to prototype wearable AI assistants. The public taxonomy, ontologies, and de-identification pipeline also provide reusable infrastructure for regulated medical AI products.

Future Directions

• Improving multimodal fusion. Early fusion provided no gain in classification and only modest gains in segmentation; developing late-fusion, cross-attention, or modality-specific temporal models could unlock the complementary value of video, audio, and IMU.

• Closing the sim-to-real gap. Models trained on simulated data may underperform in the field due to distributional shifts in environment and behavior; transfer learning, domain adaptation, and multimodal redundancy are proposed as mitigations.

• Multiperson and multiview modeling. The dataset includes synchronous annotations from secondary responders and multiple views, but the current benchmarks only use the primary responder's egocentric stream — future work could exploit team coordination dynamics.

• Ethical and governance frameworks for real-world deployment. Open questions remain about obtaining consent from unconscious patients, capturing bystanders and minors, handling sensitive situations (domestic violence, substance use), and achieving HIPAA-compliant on-device de-identification.

• Zero-shot and foundation model capabilities. Qwen-2.5's surprisingly strong zero-shot performance (38.3% classification, 55.5% segmentation) hints at a path toward general-purpose EMS understanding without extensive labeling.

Target Audience

This paper is most valuable to researchers in egocentric vision, multimodal activity recognition, and human activity understanding who need realistic, safety-critical datasets to test their methods. It also targets AI-for-healthcare and clinical decision-support researchers, AR/wearable computing developers building responder-facing assistants, and EMS training and simulation specialists interested in objective skill assessment. Data governance and medical ethics researchers will also find the IRB framework, de-identification pipeline, and ethical discussion useful when designing similar privacy-sensitive datasets.

Authors’ abstract

Emergency Medical Services (EMS) are critical to patient survival in emergencies, but first responders often face intense cognitive demands in high-stakes situations. AI cognitive assistants, acting as virtual partners, have the potential to ease this burden by supporting real-time data collection and decision making. In pursuit of this vision, we introduce EgoEMS, the first end-to-end, high-fidelity, multimodal, multiperson dataset capturing over 20 hours of realistic, procedural EMS activities from an egocentric view in 233 simulated emergency scenarios performed by 62 participants, including 46 EMS professionals. Developed in collaboration with EMS experts and aligned with national standards, EgoEMS is captured using an open-source, low-cost, and replicable data collection system and is annotated with keysteps, timestamped audio transcripts with speaker diarization, action quality metrics, and bounding boxes with segmentation masks. Emphasizing realism, the dataset includes responder-patient interactions reflecting real-world emergency dynamics. We also present a suite of benchmarks for real-time multimodal keystep recognition and action quality estimation, essential for developing AI support tools for EMS. We hope EgoEMS inspires the research community to push the boundaries of intelligent EMS systems and ultimately contribute to improved patient outcomes.