RFM-HRI : A Multimodal Dataset of Medical Robot Failure, User Reaction and Recovery Preferences for Item Retrieval Tasks

AI summary

Overview

Research area: Human–robot interaction (HRI), multimodal affective computing, and healthcare robotics — specifically robot failure and recovery during item retrieval tasks.

Technical level: Intermediate. The paper is readable without deep machine-learning background, but it assumes familiarity with standard HRI and affective-computing vocabulary (Wizard-of-Oz protocols, Facial Action Units, Self-Assessment Manikin, action-unit/gaze/pose sensing).

Scope: The paper introduces RFM-HRI, a publicly released multimodal dataset of 41 participants interacting with a crash cart robot across four systematically injected failure types (speech, timing, comprehension, search), plus one success condition, recorded in both laboratory and hospital settings.

Note: the paper text supplied here is truncated partway through Section 3.3.4, so the detailed dataset-statistics and analysis sections mentioned in the abstract are summarized only to the extent the abstract and introduction report them.

What This Paper Is About

Robots deployed in real settings will inevitably fail at some point during an interaction, and how people react — verbally and non-verbally — is poorly understood, especially in healthcare, where a failure can affect task performance and trust. Existing failure datasets mostly come from conversational or social settings, not from physically grounded, time-critical work like retrieving supplies from a crash cart. The authors build a controlled study that deliberately induces four types of communication failures on a crash cart robot and record what participants say, how their faces and heads move, and how they later rate their emotions and preferred recovery strategies, then release the resulting dataset.

Key Contributions

1. RFM-HRI, a public multimodal dataset. 214 interaction samples from 41 participants (29 in lab, 12 in hospital), each including synchronized facial action units, head pose, speech transcriptions, and post-interaction self-reports of emotional reaction and preferred robot recovery strategy.
2. A defined and evaluated set of robot failure modes. Four failure types — speech, timing, search, and comprehension — derived from three years of empirical observation of crash cart robot interactions in emergency medicine contexts, together with analysis of how users respond to each type and its associated recovery strategy.
3. A new failure-and-recovery scenario. A crash cart robot guiding item retrieval, providing a realistic embodied task in which verbal and non-verbal recovery strategies can be systematically compared for safety-critical HRI.
4. Design implications for failure-to-recovery loops. Evidence that users strongly prefer verbal recovery strategies, and that emotional responses shift over repeated failures, with implications for how robots should detect and repair breakdowns.

Main Findings

• Failures lower mood and control: Failure trials "significantly degrade affective valence" and "reduce perceived control" compared with successful interactions, as measured by Self-Assessment Manikin ratings of valence, arousal, and perceived control.
• Failure-associated emotions: Failures are strongly associated with confusion, annoyance, and frustration.
• Success-associated emotions: Successful interactions are characterized by surprise, relief, and confidence in task completion.
• Emotions change over repeated failures: Across repeated failures, confusion decreases while frustration increases over time.
• Recovery preferences are overwhelmingly verbal: Participants showed a strong preference for verbal recovery strategies; multimodal and nonverbal strategies were selected far less frequently. Example strategies offered included an explicit apology and clarification, more step-by-step guidance, and escalation to a human teammate.
• Four failure types were induced: Speech failure (under-specified instruction, e.g., "Open the drawer" without naming which of the five drawers), timing failure (a deliberate 3-second delay before directing the participant to the correct drawer), search failure (highlighting the wrong drawer via both speech and LED cues), and comprehension failure (the robot states it did not understand the request and asks for repetition without offering guidance).
• Setting coverage: Data were collected in a university campus laboratory (proxy items such as office and hardware supplies) and in a hospital clinical training area stocked with standard medical supplies; hospital drawer 6 was empty and the in-lab cart had no drawer 6.

Methodology in Plain English

The team ran a within-subjects Wizard-of-Oz study: participants believed the crash cart robot was operating autonomously, but a hidden human operator controlled it through a custom web interface. Each of the 41 participants completed a predetermined sequence of five trials — four containing one injected failure each and one error-free — presented in randomized order, so the same person experienced every condition and individual differences were controlled for.

The task was guided item retrieval. The robot asked the participant to fetch a specific item; the participant asked the cart for it verbally; the operator selected the item from an inventory list mapped to its physical location; and the cart responded with a spoken announcement of the drawer plus an illuminated LED strip segment marking the correct drawer. Items on the open top shelf were indicated by speech alone.

Failures were injected live through a dedicated module in the operator interface. Participants were never told failures would occur, and if someone asked whether the robot was malfunctioning, the facilitator gave a neutral scripted reply to avoid biasing their interpretation.

Sessions lasted about 30 minutes (study sessions were scheduled for roughly 20 minutes, with the first 7–8 minutes devoted to interaction trials) and had four phases: introduction and consent, the guided retrieval block, a post-study reflection survey, and an optional demographic and background survey. Because of time constraints, a small number of participants completed slightly fewer or more than five trials, producing minor imbalance across conditions that the analyses account for. In the reflection survey, participants named their dominant emotion for each trial from a fixed list (frustration, confusion, stress/pressure, annoyance, surprise, confidence, relief, curiosity, neutrality, or an open-ended option), rated valence, arousal, and perceived control on the Self-Assessment Manikin, and chose their preferred recovery strategy. The robot logged trial start and end times and which scripted failure was induced, enabling post-hoc annotation.

Participants were recruited via university campus advertisements and outreach to Emergency Medicine listservs of Registered Nurses and medical residents in the Global North using convenience and purposive sampling. Recruitment yielded 12 healthcare professionals (Registered Nurses and Residents, with 10 to 15 years of clinical experience) and 29 non-experts. All participants were compensated with a $10 gift card. Demographic responses were voluntary, and 39% provided complete responses; among those who responded, the sample included 11 male and 5 female participants, ages 22 to 44 (M = 25.8, SD = 6.4). Participants were not required to have prior experience with robotic systems.

Why This Matters

Impact on research. Most prior failure datasets are conversational or observational, and most healthcare robotics work is qualitative. RFM-HRI is positioned as the first dataset to combine a healthcare-motivated crash cart robot, controlled labeled failure injection across four failure types, and aligned multimodal social signals plus post-hoc reflection from participants who re-watched their own interactions. It also distinguishes system-perspective errors (technical breakdowns in the robot's designed functionality) from user-perspective errors (failures experienced by the user through miscommunication, even when hardware and software operate correctly) — a vocabulary that supports both failure detection and failure interpretation.

Real-world applications:

• Crash cart and supply retrieval in emergency medicine, where delays in locating medications and equipment can disrupt workflow and divert attention from patient care and team coordination.
• General embodied item retrieval and handoff in settings such as manufacturing and factory environments, where robots must retrieve, transport, and deliver objects under time pressure and partial observability.
• Affect-, gaze-, or posture-based failure detection models that can recognize when an interaction is going wrong before the user explicitly complains or aborts the task.
• Design of recovery policies for safety-critical robots, where verbal repair is the strategy users actually want.

Industry relevance. The dataset and failure taxonomy are directly useful to companies building service, logistics, and healthcare-assistive robots, and to teams training multimodal models for error detection and recovery. Because the failures are labeled at the system level while reactions are captured at the user level, the data can support both engineering-side anomaly detection and human-facing interaction design. The scenario is also framed as a reusable benchmark for comparing verbal versus non-verbal recovery strategies.

Future Directions

• Train and evaluate automatic failure-detection models on the facial action unit, head pose, and speech signals in RFM-HRI, and test whether those detectors fire early enough to trigger recovery in real time.
• Close the loop: move from offline recovery-preference analysis to testing whether the verbally preferred recovery strategies actually restore trust, perceived control, and task performance after a failure.
• Expand the participant pool and setting coverage — the current dataset has 41 participants with a minority of clinical experts (12), and demographic responses were only complete for 39% of them.
• Study repeated-failure dynamics over longer horizons, given that confusion decreased while frustration increased across repeated failures in this study.
• Extend the failure taxonomy and retrieval scenario beyond healthcare to other time-pressured item retrieval deployments, and transition the Wizard-of-Oz control logic toward genuine autonomy.

Target Audience

HRI researchers studying robot failure, error detection, and recovery; affective-computing and multimodal machine-learning researchers who need labeled, synchronized human reaction data tied to specific robot errors; healthcare robotics and clinical-systems researchers interested in crash cart and supply-retrieval workflows; and designers or engineers building robots that must fail gracefully in safety-critical, time-pressured environments.

Authors’ abstract

While robots deployed in real-world environments inevitably experience interaction failures, understanding how users respond through verbal and non-verbal behaviors remains under-explored in human-robot interaction (HRI). This gap is particularly significant in healthcare-inspired settings, where interaction failures can directly affect task performance and user trust. We present the Robot Failures in Medical HRI (RFM-HRI) Dataset, a multimodal dataset capturing dyadic interactions between humans and robots embodied in crash carts, where communication failures are systematically induced during item retrieval tasks. Through Wizard-of-Oz studies with 41 participants across laboratory and hospital settings, we recorded responses to four failure types (speech, timing, comprehension, and search) derived from three years of crash-cart robot interaction data. The dataset contains 214 interaction samples including facial action units, head pose, speech transcripts, and post-interaction self-reports. Our analysis shows that failures significantly degrade affective valence and reduce perceived control compared to successful interactions. Failures are strongly associated with confusion, annoyance, and frustration, while successful interactions are characterized by surprise, relief, and confidence in task completion. Emotional responses also evolve across repeated failures, with confusion decreasing and frustration increasing over time. This work contributes (1) a publicly available multimodal dataset (RFM-HRI), (2) analysis of user responses to different failure types and preferred recovery strategies, and (3) a crash-cart retrieval scenario enabling systematic comparison of recovery strategies with implications for safety-critical failure recovery. Our findings provide a foundation for failure detection and recovery methods in embodied HRI.