HAROOD: A Benchmark for Out-of-distribution Generalization in Sensor-based Human Activity Recognition

AI summary

Overview

• Research area: Sensor-based human activity recognition (HAR), specifically out-of-distribution (OOD) / domain generalization for wearable and ambient time-series sensor data.
• Technical level: Intermediate. The benchmark concepts, datasets, and protocols are easy to follow, but the 16 compared algorithms assume some familiarity with domain generalization terminology.
• Scope: The paper introduces HAROOD, an open-source PyTorch benchmark that standardizes OOD evaluation in HAR across 6 datasets, 4 distribution-shift scenarios, 16 methods, 2 backbones, and 2 model-selection protocols.

What This Paper Is About

Models that recognize human activities from sensors (accelerometers, gyroscopes, IMUs) often fail when deployed on new people, new sensor positions, new devices, or later time periods, because the data distribution shifts. Existing OOD algorithms have been tried on HAR, but only in scattered settings, so it has been unclear whether OOD methods are actually needed for HAR and which one works best. The paper builds a unified benchmark to answer those questions systematically and reproducibly.

Key Contributions

1. A unified OOD-HAR benchmark. HAROOD is presented as the first unified benchmark for OOD generalization in sensor-based HAR, released as an open PyTorch testbed at https://github.com/AIFrontierLab/HAROOD with a modular design intended to make new datasets, algorithms, comparisons, and analyses easy to add.
2. Four realistic domain-shift scenarios. The authors define cross-person, cross-position, cross-dataset (also described as cross-device in parts of the paper), and cross-time shifts, and construct concrete domain splits for each across six public datasets: DSADS, USC-HAD, UCI-HAR, PAMAP2, EMG, and WESAD.
3. A large comparative study. Sixteen OOD algorithms (ERM, Mixup, DDLearn, DANN, CORAL, MMD, VREx, LAG, MLDG, RSC, GroupDRO, ANDMask, Fish, Fishr, URM, ERM++) are evaluated under both CNN-based and Transformer-based backbones, using leave-one-domain-out evaluation equivalent to 88,320 training runs.
4. Analysis beyond leaderboards. The paper adds ranking-based method comparison, class-level behavior analysis, computational cost analysis, and practical recommendations for algorithm selection.

Main Findings

• No single method wins everywhere. CORAL achieves the best performance on the cross-person DSADS task, URM performs best on UCI-HAR, and ERM achieves the highest accuracy in cross-position tasks. ERM++ dominates cross-time generalization with a CNN backbone, but its performance becomes unsatisfactory when the backbone is switched.
• Safe defaults exist. When the choice is uncertain, CORAL, Fish, and Fishr are identified as potentially good options because their results are consistent across settings.
• The backbone matters as much as the algorithm. Under a CNN backbone, ERM++ ranks first, while ANDMask ranks 14th and LAG ranks 12th. Under a Transformer backbone, ERM++ drops to 14th, while ANDMask and LAG rise into the top three. The effect is especially large for MLDG, possibly because the CNN grid search did not find optimal hyperparameters.
• Oracle model selection beats training-domain validation by nearly 2 percentage points. The ranking order across methods stays largely consistent regardless of selection strategy, with some local reorderings: ERM++ is top-ranked in both Valid-CNN and Oracle-CNN, while ERM moves from 11th to 8th.
• HAR-specific OOD methods do not transfer their reported dominance. Across all tasks, CNN-based LAG and DDLearn rank 12th and 15th, and Transformer-based LAG and DDLearn rank 3rd and 15th. The authors suggest this may stem from the modifications they made for consistency and fairness in a unified codebase, and invite authors to contribute their own implementations.
• Performance is class-dependent. Method accuracy varies across activity classes, suggesting that combining methods could improve overall performance, particularly when misclassification patterns and class-specific performance are taken into account.
• Adaptive model selection shows promise. Meta-learning or importance-weighting-style mechanisms are flagged as promising for improving performance under domain shifts.
• Distribution shifts are unevenly hard. Inter-domain distances measured with Maximum Mean Discrepancy (MMD), Wasserstein distance, and Earth Mover's Distance (EMD) under both min-max and standard-score normalization show distances driven not only by task difficulty but also by data dimensionality and number of classes. Under min-max normalization, the largest MMD value reported is 0.72886 for cross-position DSADS.
• Reported accuracy values. Numerical accuracy results are presented only in the paper's figures; specific accuracy numbers are not reported in the text provided.

Methodology in Plain English

The authors first formalize the problem: training data comes from several labeled source domains, no target-domain data is available at any point, and the target distribution is different from every source distribution. They distinguish this from domain adaptation, source-free domain adaptation, test-time adaptation, and multi-task learning, which all use some form of target information.

They then assemble six public time-series sensor datasets and, for each of the four shift scenarios, segment and normalize the raw signals and split them into separate domains. In the cross-person scenario, subjects are grouped into domains; in cross-position, sensor placements form the domains; in cross-dataset, four datasets (DSADS, USC-HAD, PAMAP2, UCI-HAR) serve as four domains; in cross-time, each time series is divided into chronological segments.

For evaluation, they follow the leave-one-domain-out protocol: train on all but one domain, test on the held-out one, repeat for every domain, and repeat the whole protocol three times with different splits and seeds. Sixteen algorithms are implemented behind a single fixed API, each running with a CNN backbone (two convolution–batch-norm–ReLU–max-pool blocks) and a Transformer backbone. Hyperparameters are tuned by grid search: data is split 80%/20% for training and validation, training runs for up to 150 epochs with Adam and a weight decay of 5×10⁻⁴, 20 hyperparameter combinations are enumerated (for ERM, learning rates {0.001, 0.005, 0.01, 0.05, 0.1} and batch sizes {32, 64, 128, 256}), and each setting is run three times — 240 training runs per algorithm in a four-domain cross-person setting such as DSADS. In addition to average test accuracy, they rank methods per task and sum the ranks to get an overall ordering.

Why This Matters

• Research impact: HAR has lacked a unifying benchmark comparable to DomainBed in computer vision, so results have been scattered across proprietary setups and limited dataset combinations, sometimes without released code or data. HAROOD provides standardized splits, a shared codebase, and identical conditions for 16 methods, which makes comparisons reproducible and reveals that several well-known OOD methods do not outperform a simple ERM baseline in HAR.
• Healthcare monitoring: the paper motivates the problem with fall detection, noting it is neither feasible nor ethical to induce large numbers of elderly people to fall just to collect training data, so models must generalize from other populations.
• Assisted living and smart homes: recognizing activities across residents, rooms, and sensor placements without retraining per deployment.
• Fitness tracking: handling differences in device hardware, wrist versus pocket placement, and individual biomechanics.
• Edge deployment: the paper argues that benchmarking small CNN and Transformer models remains valuable because they can run at the edge with minimal resources and far lower cost than LLM-based HAR approaches such as HARGPT and SensorLLM, while still supporting future upgrades to larger models.
• Industry relevance: practitioners deploying wearable or ambient sensing products need guidance on which algorithm to pick for a given type of shift and which backbone to pair with it; the paper's findings on CORAL, Fish, and Fishr as consistent options, and on the CNN-versus-Transformer trade-off, give directly actionable direction.

Future Directions

• Hybrid and task-tailored design. Since no method dominates, the authors suggest selecting algorithms and backbones based on task characteristics, and that hybrid models and more nuanced architecture design are needed.
• Better model selection. Training-domain validation selection can overfit when validation and test distributions differ, while oracle selection leaks test information; adaptive mechanisms such as meta-learning or importance weighting are proposed as promising, and intelligent hyperparameter tuning is identified as an important area.
• Class-level and ensemble approaches. Because methods differ in which activity classes they get right, combining methods to exploit complementary strengths is proposed.
• Community contributions and extension. The authors invite researchers to contribute their own implementations of HAR-specific methods (to test whether the unified-codebase results hold), and to extend the benchmark with new datasets, algorithms, comparisons, and analyses, with an eye toward future LLM-based HAR models.

Target Audience

Researchers and graduate students working on human activity recognition, wearable sensing, and domain generalization or transfer learning; benchmark builders who want a template for standardized evaluation in time-series domains; and industry practitioners deploying sensor-based activity recognition who need evidence about which OOD algorithm and backbone to use under a specific distribution shift.

Authors’ abstract

Sensor-based human activity recognition (HAR) mines activity patterns from the time-series sensory data. In realistic scenarios, variations across individuals, devices, environments, and time introduce significant distributional shifts for the same activities. Recent efforts attempt to solve this challenge by applying or adapting existing out-of-distribution (OOD) algorithms, but only in certain distribution shift scenarios (e.g., cross-device or cross-position), lacking comprehensive insights on the effectiveness of these algorithms. For instance, is OOD necessary to HAR? Which OOD algorithm performs the best? In this paper, we fill this gap by proposing HAROOD, a comprehensive benchmark for HAR in OOD settings. We define 4 OOD scenarios: cross-person, cross-position, cross-dataset, and cross-time, and build a testbed covering 6 datasets, 16 comparative methods (implemented with CNN-based and Transformer-based architectures), and two model selection protocols. Then, we conduct extensive experiments and present several findings for future research, e.g., no single method consistently outperforms others, highlighting substantial opportunity for advancement. Our codebase is highly modular and easy to extend for new datasets, algorithms, comparisons, and analysis, with the hope to facilitate the research in OOD-based HAR. Our implementation is released and can be found at https://github.com/AIFrontierLab/HAROOD.