A Smart-Glasses for Emergency Medical Services via Multimodal Multitask Learning

AI summary

Overview

Research area: Machine learning systems for healthcare — specifically multimodal, multitask deep learning combined with edge/on-device inference serving on wearable hardware for Emergency Medical Services (EMS).

Technical level: Intermediate to Advanced. The paper assumes familiarity with multimodal encoders, model serving, feature caching, quantization-adjacent offloading decisions, and clinical EMS workflows.

Scope in one sentence: The paper builds and evaluates EMSGlass, a smart-glasses system pairing the EMSNet multimodal multitask model (text, vitals, scene images) with the EMSServe caching-and-offloading serving framework, reporting accuracy against unimodal baselines, a 1.9×–11.7× serving speedup over direct PyTorch inference, and a six-EMT user study.

What This Paper Is About

Emergency Medical Technicians must make rapid, life-critical decisions while under heavy cognitive load, and the profession suffers severe workforce shortages from stress and burnout. Existing smart-glasses EMS assistants (e.g., EMSAssist, CognitiveEMS) only use patient symptom speech, ignoring vital signs and scene images, and rely on serving frameworks that assume all data modalities arrive at the same time. This paper's goal is to unify text, vitals, and scene imagery into a single real-time model that handles five EMS tasks at once, serve it fast enough for asynchronous real-world data arrival, and validate the design with working EMTs.

Key Contributions

1. EMSNet — the first multimodal multitask model trained on large real-world multimodal EMS datasets, simultaneously supporting five critical EMS tasks: protocol selection, medicine type recommendation, medicine quantity recommendation, dosage recommendation, and disease history inference.
2. EMSServe — a serving framework for multimodal models that handles differing modality arrival times, using a modality-aware model splitter, feature caching, model inference time profiling, and adaptive edge-assisted offloading across glass and edge server. It reports 1.9×–11.7× speedup over direct PyTorch multimodal inference across diverse edge devices (Google Glass) and edge servers.
3. A multimodal EMS data processor — described as the first tool of its kind to prepare multimodal EMS datasets, and the first to prepare the medicine-type and medicine-quantity tasks for EMS, built on the NEMSIS 2023 database.
4. A seamless on-display user interface and comprehensive user study — continuous recommendation updates on the Google Glass display with lower latencies, evaluated with six professional EMTs, yielding both quantitative and qualitative findings.

The paper states that its data, code, and model will be open-sourced upon acceptance.

Main Findings

• Three identified gaps motivate the work. Challenge #1: no multimodal model supports multiple simultaneous EMS tasks — SOTA systems use only patient symptoms and ignore vitals and scene images, and focus only on protocol selection while EMTs also need treatment personalization. Challenge #2: existing serving frameworks (PyTorch, TensorFlow) assume simultaneous modality availability, causing redundant recomputation when modalities arrive asynchronously. Challenge #3: existing user interfaces ignore EMT mobility and variable data arrival times, and prior works lack real-world user studies with end-to-end EMT usage.

• Vitals can arrive many times per event. Real-world NEMSIS 2023 data records up to 30 vitals per event, which in a conventional multimodal framework could force the text submodule to run up to 30 times.

• Text modules dominate inference time. Across four hardware platforms, text modules (speech-to-text plus text encoder) are the primary inference bottleneck; Whisper variants require significantly more time than other components, and Whisper-small and Whisper-medium are entirely unusable on the PH1 phone and Google Glass due to memory limitations.

• Inference latency varies sharply by hardware. YOLO11n inference takes 3.2s on Google Glass, 0.7s on PH1, 0.08s on Edge-4C, and 0.03s on Edge-64X. Offloading YOLO11n from Google Glass to Edge-64X achieves over 100× speedup.

• Speech recognition degrades on wearable hardware. Whisper-t's word error rate rises from 0.139 (HyperX microphone) to 0.315 (Google Glass). Google Glass imposes an 8kHz cutoff, and spectrograms show lost high-frequency signals. The paper attributes poor generalization to model size — all prior models are under 100 million parameters (EMSConformer: 10m, Whisper-tiny: 74m, Whisper-base: 74m) — and proposes Whisper-small (242m) and Whisper-medium (764m) instead.

• Open-set object detectors are unsuitable for EMS objects despite high recall. Grounding DINO (Swin-T: 172m, Swin-B: 341m) shows mAP below 0.2 but recall above 0.55, indicating excessive false positives, which the authors argue limits its suitability where precision is crucial. A dedicated close-set detector (YOLO11) is trained instead.

• Human-in-the-loop annotation halves annotation effort. Using Grounding DINO to auto-label, then having humans adjust those labels before fine-tuning YOLO11, reduced annotation time by half — on average from 10 to 5 minutes for every 100 images.

• Feature caching and offloading yield the headline serving result. EMSServe's modality-aware splitter decomposes a multimodal model into single-mode modules so text features can be computed and cached before vitals arrive. Text feature caching is done in parallel (hiding its cost behind the mandatory text model run), while vitals feature caching is done serially because vitals compute is up to four orders of magnitude cheaper and parallel thread launch costs more.

• Fault tolerance is designed in. When the edge server returns inference and recommendation results, it also returns the computed feature cache, so the glass cache is never outdated by more than one step; if the edge server crashes during cache computation, the glass switches seamlessly to on-device inference.

• 2-modal results on dataset D1 (text, vitals), per the truncated Table 3. Vitals-only baselines using LSTM, RNN, and GRU reach protocol-selection top-1/3/5 of 0.43/0.66/0.76, 0.44/0.68/0.77, and 0.45/0.68/0.78 respectively. Text-only baselines such as MobileBERT reach 0.77/0.94/0.97, TinyBERT 0.77/0.93/0.96, and BERTBase 0.76/0.93/0.97. Combined text-vitals configurations reach up to 0.77/0.94/0.97. The full multimodal portion of the table is truncated in the supplied content, so complete multimodal results are not reported here.

• The abstract reports the overall claim: EMSNet supports up to five EMS tasks with superior accuracy compared to state-of-the-art unimodal baselines, and a user study with six professional EMTs shows EMSGlass enhances real-time situational awareness, decision-making speed, and operational efficiency.

Methodology in Plain English

The researchers started from NEMSIS, a national database of real-world emergency event reports, and wrote a plug-and-play pipeline to turn its records into four datasets. D1 pairs symptom text with time-series vitals (123,803 samples); D2 adds one-hot scene information about alcohol and pills (3,005 samples, a disjoint set of events); D3 supplies audio for speech-to-text training (1123 training and 600 validation samples, plus 500 test samples recorded by five users on a HyperX Solocast and Google Glass); and D4 supplies scene images for object detection (1,340 samples recorded on Google Glass across five different rooms).

EMSNet has three encoder modules — a text encoder over symptom descriptions, a vitals encoder over time-series vitals such as BP, HR, PO, RR, CO2, and BG, and an image encoder over detection output. Their vectors are concatenated into one unified feature representation that feeds three headers: protocol selection, medicine type, and medicine quantity. When a medicine bottle is detected, OCR and a barcode scanner read the label, a division-based Med-Math step turns quantity plus concentration into a dosage recommendation, and the medicine name is mapped to a list of 82 common EMS diseases for disease-history inference.

Because the 3-modal dataset (3,005 samples) is far smaller than the 2-modal dataset (123,803 samples), the authors use progressive modality integration: rather than training the 3-modal model from scratch, they feed the text and vitals parts of each 3-modal sample through the already-trained 2-modal model and only train the newly initialized scene encoder.

On the systems side, they profiled every module's inference time on Google Glass, a phone, and two edge servers, then split the multimodal model into single-mode modules. A heartbeat monitor measures file-transfer latency roughly every second and decides per module whether to offload to the edge or run on-glass, while cached text features eliminate repeated text-module runs. The whole thing is packaged as an Android app on Google Glass Enterprise Edition 2 (over 3,000 lines of Java) and a Torch Serve-based backend on the Edge-4C server (about 2,000 lines of Python), communicating over HTTPS and WiFi.

Why This Matters

Impact on research: The paper argues it is the first to bring multimodal multitask learning to the EMS domain, and the first to write a data processor that prepares multimodal EMS datasets — including the first preparation of medicine-type and medicine-quantity tasks from NEMSIS. It also reframes a systems problem: asynchronous modality arrival is treated as a first-class serving constraint, motivating modality-aware splitting, feature caching, and offloading decisions rather than assuming batch-synchronous inputs.

Real-world applications:

• Pre-hospital emergency response in disasters, such as the California wildfires in January 2025, where cloud connectivity is unreliable and hands-free operation matters.
• Family emergency care in rural areas, where EMS resources and specialty support are scarce.
• Overdose and respiratory emergencies, where the national EMS dataset mandates reporting pill and alcohol presence and accurate detection can narrow protocol options from over 100 to fewer than 5.
• Medication administration, where OCR and barcode scanning can automate drug name and concentration extraction and reduce Med-Math errors — the paper's example is computing 21mg ÷ 4.2mg/ml = 5ml of an adrenaline solution.

Industry relevance: The results speak directly to edge-AI and wearable-device vendors, medical device makers, and EMS software providers. The finding that latency varies dramatically across hardware and that offloading one detector yields over 100× speedup gives a concrete engineering argument for hybrid glass-plus-edge architectures. The user study's qualitative insights are framed as actionable directions for manufacturers and system designers pursuing next-generation AI-enabled EMS.

Future Directions

• Scaling speech models further. The authors chose Whisper-small (242m) and Whisper-medium (764m) as one order of magnitude above prior SOTA, and explicitly note that further exploration of alternative scaling options is future work.
• Extending EMSNet beyond the current five tasks. The paper frames the qualitative user-study insights as directions for extending EMSGlass toward next-generation AI-enabled EMS systems, implying additional tasks or modalities.
• Handling edge-server failure and connectivity more robustly. The current fault-tolerance design keeps cache staleness to one step and falls back to on-device inference, but the conditions under which that fallback remains acceptable in the field are left open.
• Broadening real-world validation. The evaluation rests on a user study with six professional EMTs; larger end-to-end field deployments would test whether interaction designs hold under genuine EMS conditions.

Target Audience

This paper is most useful to researchers in multimodal and multitask learning who care about deployment constraints rather than benchmark-only accuracy; systems researchers working on edge/on-device inference, model splitting, and serving frameworks; wearable and medical-device engineers; and clinical informatics and EMS professionals evaluating whether AI assistants fit real pre-hospital workflows. Readers who want the data-processing pipeline specifically — building multimodal datasets from tabular national registries like NEMSIS — will also find the dataset construction sections directly relevant.

Authors’ abstract

Emergency Medical Technicians (EMTs) operate in high-pressure environments, making rapid, life-critical decisions under heavy cognitive and operational loads. We present EMSGlass, a smart-glasses system powered by EMSNet, the first multimodal multitask model for Emergency Medical Services (EMS), and EMSServe, a low-latency multimodal serving framework tailored to EMS scenarios. EMSNet integrates text, vital signs, and scene images to construct a unified real-time understanding of EMS incidents. Trained on real-world multimodal EMS datasets, EMSNet simultaneously supports up to five critical EMS tasks with superior accuracy compared to state-of-the-art unimodal baselines. Built on top of PyTorch, EMSServe introduces a modality-aware model splitter and a feature caching mechanism, achieving adaptive and efficient inference across heterogeneous hardware while addressing the challenge of asynchronous modality arrival in the field. By optimizing multimodal inference execution in EMS scenarios, EMSServe achieves 1.9x -- 11.7x speedup over direct PyTorch multimodal inference. A user study evaluation with six professional EMTs demonstrates that EMSGlass enhances real-time situational awareness, decision-making speed, and operational efficiency through intuitive on-glass interaction. In addition, qualitative insights from the user study provide actionable directions for extending EMSGlass toward next-generation AI-enabled EMS systems, bridging multimodal intelligence with real-world emergency response workflows.