OMG-Bench: A New Challenging Benchmark for Skeleton-based Online Micro Hand Gesture Recognition

AI summary

Overview

Research area: Computer vision for human-computer interaction, specifically skeleton-based online gesture recognition for VR/AR systems.

Technical level: Advanced. The paper assumes familiarity with graph convolutional networks, transformer architectures, query-based detection, CTC loss, and streaming inference pipelines.

Scope: The paper introduces the first large-scale public dataset for recognizing subtle "micro" hand gestures from streaming hand-skeleton data and proposes a transformer-based baseline that outperforms prior methods on it.

What This Paper Is About

Existing hand-gesture datasets for VR/AR contain large-amplitude "macro" gestures that tire users' arms, are captured with noisy single-view sensors, and present gestures in isolation rather than back-to-back. There is no public dataset for subtle, rapid "micro" gestures performed between the thumb and other fingers, and no algorithm specifically designed for recognizing them in a continuous stream. The paper builds such a dataset (OMG-Bench) and a matching end-to-end recognition model (HMATr).

Key Contributions

1. OMG-Bench dataset. The first publicly available large-scale skeleton-based online micro gesture benchmark: 40 fine-grained, easily confusable gesture classes, 1,272 sequences, 13,948 frame-level annotated instances, and 18 subjects. Skeletons are produced by a calibrated five-camera RGB-D rig with a self-supervised multi-view pose estimator (2.78 mm mean joint error) and labeled through a semi-automatic heuristic-plus-expert pipeline.

2. HMATr architecture. An end-to-end framework that unifies gesture detection and classification using a hierarchical memory bank (frame-level for fine details, window-level for semantics) and learnable position-aware queries, enabling accurate inference with non-overlapping sliding windows.

3. Query-based CTC loss. A training objective that enforces temporal alignment between predicted gestures and ground truth, counteracting the tendency of CTC-style models to collapse to blank predictions on weak micro-gesture signals.

4. Comprehensive benchmark. Re-implementations and comparisons of state-of-the-art offline, CTC-based, and boundary-supervision methods on OMG-Bench, plus generalization results on SHREC'21 and SHREC'22.

Main Findings

• Large margin over prior methods: HMATr reaches 89.2% Detection Rate, 0.22 False Positive score, 0.71 Jaccard Index, and 0.77 Normalized Levenshtein Distance, improving DR by 7.6 points over the strongest baseline (Boundary Regularization at 81.6%).

• Fastest and lowest-latency: 1.61 ms inference time and 7.67-frame average delay, both best in the comparison — a direct consequence of using non-overlapping windows plus memory rather than costly overlapping windows or two-stage pipelines.

• Micro gestures are much harder than macro gestures: OMG-Bench's mean gesture duration is 0.57 s versus 2.60 s (SHREC'21) and 1.19 s (SHREC'22); mean gesture interval is 0.22 s; normalized mean joint displacement is only 8.95 versus 158.38 and 128.73. It also has 27.60% same-class continuous gestures compared with under 0.3% in prior datasets.

• Both memory levels matter: Removing the frame-level memory drops DR to 86.1%; removing the window-level memory drops it to 87.0%; removing position-aware queries drops it to 82.4%. The Position-aware Interaction module is the single most important interaction block (removal costs 6.8 DR points).

• Generalizes to macro gestures: HMATr also achieves state-of-the-art results on SHREC'22 (0.95 DR) and SHREC'21 (0.92 DR), showing the design is not micro-gesture-specific.

• Distinct query roles emerge: Visualization shows different learnable queries specialize to different temporal positions in the window, implicitly diversifying gesture feature representations.

• Better boundary handling: Qualitative results show HMATr segments consecutive same-class gestures correctly, whereas baselines tend to merge or over-segment them.

Methodology in Plain English

The team built a five-camera rig that captures hands from multiple angles at 30 FPS. Instead of hand-labeling skeletons, they run a self-supervised multi-view pose estimator that fuses depth images across cameras into accurate 21-joint hand skeletons. A rule-based recognizer based on finger-joint distances generates rough start/end timestamps for each gesture, and five human experts then review and correct those boundaries — a semi-automatic labeling scheme that keeps quality high without manual annotation of every frame.

For the recognition model, the data arrives as a continuous stream that is chopped into fixed, non-overlapping windows. Because non-overlapping windows can split a gesture across a boundary and lose context, the model keeps two memory queues: one storing recent frame-level features (fine motion detail) and one storing recent window-level query features (higher-level semantics). Both are FIFO queues updated as new windows arrive. A set of learnable "queries" — abstract slots that the model trains to represent potential gestures — are initialized from the window-level memory so they carry historical context, then interact via attention with both the frame memory and the current window. Each query finally predicts a gesture class and a position (center and width) inside the window. Training uses bipartite matching to assign queries to ground-truth gestures, a classification and position loss on matched pairs, and a query-based CTC loss that prevents the network from defaulting to "no gesture."

Why This Matters

Research impact. Micro-gesture recognition from skeletons has had no public benchmark to build on, so results were not comparable across groups. OMG-Bench creates a standardized, difficult evaluation target, and HMATr provides a strong reproducible baseline. The hierarchical-memory idea is also relevant beyond gestures, to any streaming action-recognition problem with short, subtle events.

Real-world applications:

• VR/AR menu selection and text entry using small thumb-to-finger taps, reducing arm fatigue during long sessions.
• Silent input for smart glasses and head-mounted displays where large arm motions are socially or physically impractical.
• Accessibility interfaces for users with limited hand mobility who cannot perform large gestures.
• Gesture-based control in automotive, medical, or industrial settings where hands-free, low-effort interaction is valuable.

Industry relevance. Meta Quest, PICO, and similar head-mounted displays already ship hand tracking, so a skeleton-based micro-gesture pipeline can be deployed without new hardware. HMATr's 1.61 ms inference and low latency are directly relevant to real-time interaction budgets, and the non-overlapping window design reduces compute relative to overlapping-window alternatives.

Future Directions

• Expanding modality coverage: The dataset uses only skeletons; fusing depth, surface EMG, or radar could resolve the most confusable classes.
• Bimanual and two-handed gestures: The current 40 classes are all single-hand, thumb-to-finger interactions; two-handed micro gestures remain unexplored.
• Cross-user and cross-device generalization: The current split is cross-subject with one capture rig — robustness to different hand sizes, skin tones, camera placements, and pose-estimator errors is untested.
• Joint optimization with gesture semantics: Whether micro gestures should be designed around a usable command vocabulary, and how detection latency interacts with downstream UI feedback loops, are open system-level questions.

Target Audience

Researchers and engineers working on gesture recognition, skeleton-based action understanding, egocentric/VR interaction, and streaming video or sensor inference. It is most useful to those already comfortable with graph convolutional networks and transformer attention, and to practitioners who need a benchmark to evaluate a new online gesture model or a dataset to train a production micro-gesture interface.

Authors’ abstract

Online micro gesture recognition from hand skeletons is critical for VR/AR interaction but faces challenges due to limited public datasets and task-specific algorithms. Micro gestures involve subtle motion patterns, which make constructing datasets with precise skeletons and frame-level annotations difficult. To this end, we develop a multi-view self-supervised pipeline to automatically generate skeleton data, complemented by heuristic rules and expert refinement for semi-automatic annotation. Based on this pipeline, we introduce OMG-Bench, the first large-scale public benchmark for skeleton-based online micro gesture recognition. It features 40 fine-grained gesture classes with 13,948 instances across 1,272 sequences, characterized by subtle motions, rapid dynamics, and continuous execution. To tackle these challenges, we propose Hierarchical Memory-Augmented Transformer (HMATr), an end-to-end framework that unifies gesture detection and classification by leveraging hierarchical memory banks which store frame-level details and window-level semantics to preserve historical context. In addition, it employs learnable position-aware queries initialized from the memory to implicitly encode gesture positions and semantics. Experiments show that HMATr outperforms state-of-the-art methods by 7.6% in detection rate, establishing a strong baseline for online micro gesture recognition. Project page: https://omg-bench.github.io/