HUI360: A 360° Egocentric Dataset and Baselines for Human-Robot Interaction Anticipation

AI summary

Overview

Research area: Computer vision and human-robot interaction (HRI), specifically egocentric 360° perception, human pose/segmentation annotation pipelines, and interaction anticipation benchmarks.

Technical level: Intermediate. The paper assumes familiarity with object detection and tracking, 2D pose estimation, segmentation, and classification metrics (AUC, AP, F1), but its core contributions are dataset construction and benchmarking rather than novel network architecture.

Scope: The paper introduces HUI360, a large in-the-wild 360° egocentric dataset from a mobile robot with an automatic interaction-annotation pipeline, plus baselines for anticipating when a person will physically interact with that robot.

What This Paper Is About

Robots that operate around people need to predict whether a passerby is about to interact with them, so they can respond proactively. Existing HRI datasets for this task are described in the paper as too small, too uniform in context, tied to specific hardware, or not capturing spontaneous behavior in the wild, and there is no shared evaluation protocol or cross-dataset comparison. The authors address this by releasing a large, diverse, curated dataset (HUI360), an open-source pipeline that annotates interactions in arbitrary 360° videos, and a formalized anticipation benchmark with baselines, including the first cross-dataset evaluations for this task.

Key Contributions

1. HUI360 dataset: described as the largest dataset for human-robot interaction anticipation in the wild, recorded from a mobile robot over 20 days within a 3-month period, across 9 environments, yielding 68 recordings, 1,937 episodes, 71.3 hours of recording (11.3 hours of episodes at 15 fps, 612,378 frames), 30 setups, 4,310 tracks and 375 interacting tracks (9%). Raw panoptic 360° images are released on demand for research purposes only, in compliance with GDPR.
2. An open-source automatic annotation pipeline: detection, filtering, segmentation, tracking and interaction labeling for arbitrary 360° equirectangular videos, with interfaces for manual refinement. Using it, the authors release an open set of 1M pre-processed annotations including 2D poses, facial keypoints and segmentation masks.
3. SSUP-A, a cross-domain extension: 6M annotations generated by applying the pipeline to the existing in-the-wild outdoor dataset SSUP-HRI, totaling 1,149,438 frames, 27,698 tracks and 419 interacting tracks (2%) across two field sessions of 5 days each in New York public squares. This enables cross-dataset transfer evaluation.
4. A formalized evaluation protocol and baselines: an interaction-anticipation formulation with an advance-detection threshold T_ADV, plus Random Forest, MLP and LSTM baselines evaluated on transfer capability, forecasting horizon, input frequency robustness, and feature ablations.

Main Findings

• LSTM is the strongest baseline. In Table III (AUC, T_ADV = 15), the LSTM reaches 0.91 on HUI360 Test when trained on HUI360 Train, versus 0.86 for MLP and 0.81 for Random Forest. On SSUP-A Test with SSUP-A Train, the LSTM reaches 0.88 versus 0.85 for MLP and 0.86 for RF.
• Cross-dataset transfer hurts performance. Training on HUI360 and testing on SSUP-A Test gives AUC 0.84 (LSTM), 0.79 (MLP), 0.83 (RF), lower than HUI360's in-dataset 0.91. Training on SSUP-A and testing on HUI360 Test gives AUC 0.84 (LSTM), 0.81 (MLP), 0.76 (RF).
• Average Precision and F1 expose class imbalance. AP on SSUP-A Test reaches only 0.16 for the LSTM trained on HUI360 and 0.24 trained on SSUP-A; F1 on SSUP-A Test ranges from 0.06 to 0.26. The positive/negative ratio is 0.19 in HUI360 Test and 0.03 in SSUP-A Test.
• Anticipation degrades as the horizon grows. Across T_ADV = 5, 10, 15, 20, 25, 30 frames (0.33 to 2.0 seconds), LSTM AUC falls from 0.97 to 0.77, MLP from 0.96 to 0.76, and RF from 0.9 to 0.71. RF degrades less gradually than MLP and LSTM according to the authors' description.
• Lower input frame rates reduce accuracy sharply at the extreme. With T^org = 30 (2 seconds), AUC at 15 Hz / 5 Hz / 3 Hz / 1 Hz is 0.91 / 0.87 / 0.85 / 0.78 for the LSTM, 0.87 / 0.87 / 0.85 / 0.75 for the MLP, and 0.81 / 0.80 / 0.80 / 0.73 for the RF, with "pronounced collapse" at one frame per second.
• More keypoints do not mean better results. The ablation over feature sets D1–D7 shows Sapiens' much more detailed keypoint sets (D4 with 242 facial keypoints, D5 with 308 keypoints) yield results similar to ViTPose for the tested databases. LSTM: 0.79 (D1), 0.81 (D2), 0.91 (D3), 0.88 (D4), 0.90 (D5), 0.90 (D6), 0.89 (D7). MLP: 0.81, 0.83, 0.87, 0.88, 0.88, 0.87, 0.89. RF: 0.78, 0.80, 0.81, 0.81, 0.81, 0.81, 0.80.
• Interaction labels are sparse. Only 9% of HUI360 tracks and 2% of SSUP-A tracks are labeled as interacting; HUI360 averages 1.7 tracks per frame (maximum 12) while SSUP-A averages 5.5 tracks per frame (maximum 19).
• Annotation quality required human curation. 15% of episodes were flagged and not exported during manual review, for reasons including abnormal novelty-driven behavior, operator presence, unrecoverable tracking issues, and removal requests under the ethics protocol and GDPR.

Methodology in Plain English

The team mounted a 360° camera on a custom-made service robot that stayed still while offering objects such as food, stickers and pins, and recorded passersby across 9 environments (Bulle12X, Cafeteria, CoffeeB, ECBack, MainEntrance, MainHallway, Room005, ECFace, Room104). No instructions were given about how to behave in front of the robot, so behaviors were spontaneous. An episode begins 10 seconds before a person is detected in front of the camera and ends 10 seconds after detection stops; recordings run from 15 minutes to 4 hours and contain 3 to 134 episodes each.

For annotation, persons are detected with YOLOv11x and tracked and segmented across an episode with SAM2.1-L. Because off-the-shelf models are not trained on equirectangular panoptic images, detection, filtering, segmentation and tracking are performed on crops ("region-images"), using 4 overlapping fixed regions with wrapping. Filtering removes low-quality detections based on box size, mask size and the number of valid visible pose keypoints. Two pose estimators are used because they are complementary: ViTPose-B (17 COCO keypoints, working at 256 × 192) and Sapiens-0.6B-Pose-308 (308 keypoints including 242 facial keypoints, working at 1024 × 768). In this dataset 26% of boxes have height below 256, 73% fall in [256, 1024] and 0.4% exceed 1024, so most people are downscaled for ViTPose and upscaled for Sapiens; 73% of ViTPose keypoints and 62% of Sapiens keypoints (64% facial, 55% other) are valid with score above 0.5.

An "interaction" is defined strictly as physical contact: a person is interacting when their segmentation mask intersects a predefined interaction zone (the robot's plate in HUI360, the trashcan openings in SSUP-A). This is fixed in the camera frame for HUI360, but the moving trashcan cameras of SSUP required a per-frame intersection mask obtained with SAM2 from a manual prompt of the initial position and refined via Convex Hull.

The anticipation task itself is framed as binary classification: given features from an observation window of T frames before an event, predict whether an interaction is coming. The authors introduce T_ADV (advance detection time), with positives labeled when the observation window ends T_ADV before the interaction onset, and windows too close to the interaction discarded. For negatives, they avoid random sampling and instead place T_0 at the moment the tracked subject appears largest in the field of view, as a proxy for distance; this avoids trivial "moving away" negatives and mitigates the imbalance from overwhelmingly non-interacting tracks. Three baselines are trained on segmented person area, bounding box position and ViTPose keypoints: a Random Forest (depth 5, 500 trees), an MLP (2 hidden ReLU layers, sizes [32, 256], 9k to 0.9M parameters at T = 30) and a 3-layer LSTM (hidden size 128, 350k to 0.85M parameters). Both neural models train for 25 epochs with weighted binary cross-entropy at a fixed learning rate of 0.001 and no dropout. AUC is the primary metric, with F1 and Average Precision reported for the main cross-dataset evaluation.

Why This Matters

Impact on research. The paper argues that a fundamental problem in the field is the lack of reproducible, comparable references and domain-transfer evaluations. By pairing a large curated dataset with a released annotation pipeline, a formalized evaluation protocol, and the homogeneous re-annotation of an existing dataset, it gives the community a shared benchmark for interaction anticipation — including the first cross-dataset evaluations reported for this task. The comparison table frames HUI360 as the only listed dataset combining in-the-wild collection, 360° field of view, 9 scenes, 2D pose, video and curation.

Real-world applications.

• Personal and service robots in public or private spaces that proactively approach people who intend to interact, improving user experience and mission efficiency.
• Mobile robots that adapt behavior without waiting for retraining when moved to a new site, which the authors connect to mobile deployments such as SSUP.
• On-board systems with limited computation, informed by the reduced-frame-rate baselines at 15, 5, 3 and 1 Hz.
• Safety and social-awareness functions where anticipating an imminent physical interaction allows the robot to slow, stop or yield.

Industry relevance. The reduced-frame-rate and variable-horizon results speak directly to deployment constraints, since the authors note on-board computational power is inherently limited. The dataset being collected in public, multi-user spaces over months, with ethics approval (Anonymous Institution Ethics Committee, June 17, 2025, decision no. 465), informed-consent signage, GDPR withdrawal handling, and on-demand release of raw 360° imagery, gives a template for how industry deployments can collect and share human data responsibly.

Future Directions

• Better architectures. The authors describe the current baselines as "architecturally basic" and point toward richer temporal and spatial modeling.
• Exploiting equirectangular imagery. The panorama's specific geometry is named as an underused direction, along with incorporating social and group dynamics.
• Broader definitions of interaction. The physical-contact definition captures only a subset of interaction behavior and does not model pre-contact signals such as gaze, hesitation, verbal engagement, or approach-and-stop behaviors; the released raw 360° data and pipeline are intended to support extensions toward more nuanced, socially grounded labels.
• Generalization across embodiments and sites. The drop observed when the robotic platform changes raises the open question of how to make anticipation methods transfer across robot size, mobility characteristics and functional role.

Target Audience

Researchers and engineers working on socially aware robotics, egocentric and 360° perception, and human motion/intent forecasting, as well as anyone needing a benchmark for interaction anticipation with cross-dataset evaluation. It is also relevant to practitioners building annotation pipelines for human-centric video at scale, and to teams who need to weigh annotation accuracy, class imbalance and on-board compute constraints in real deployments.

Authors’ abstract

As robots increasingly operate in human-populated environments, anticipating human intentions is essential for enabling proactive and socially aware behavior. Automatic anticipation of human-robot interactions is thus emerging as a crucial perception challenge for embodied agents. To this end, we introduce HUI360, the largest dataset for human-robot interaction anticipation in the wild and its set of baselines. The dataset was collected from a mobile robot, in the wild, over multiple days within a 3-month period, and in several environments, capturing natural, spontaneous behaviors from both passersby and users, and encompassing a diverse range of individuals. This variety enables evaluating and improving the generalization capabilities of interaction anticipation models. We designed a pipeline and share code for automatic interaction annotation in arbitrary 360-degree equirectangular videos, along with interfaces for manual refinement. Using this pipeline, we release the HUI360 open set of 1M pre-processed annotations, including detailed 2D poses, facial keypoints, and segmentation masks, obtained using state-of-the-art computer vision methods and manually curated to ensure high-quality tracking and interaction annotation. Additionally, we release the raw panoptic 360-degree images captured from the robot's egocentric viewpoint (on demand, for research purpose only in compliance with GDPR). Finally, we establish benchmark baselines for interaction anticipation, including the first cross-dataset evaluations for this task: to this end, we also release 6M annotations for another existing in-the-wild outdoor dataset collected from a mobile robot (SSUP-HRI). Dataset and code can be found at https://hucebot.github.io/hui360.