HiPHI: A Large-Scale Benchmark for High-Precision Human Motion and Object-Interaction

AI summary

Overview

• Research area: Robotics / humanoid learning, with connections to motion capture, human-object interaction (HOI) datasets, and reinforcement-learning-based motion imitation.
• Technical level: Intermediate.
• Scope: A large-scale, high-precision optical motion capture dataset and benchmark (617.5 hours) intended as reference motion for training, evaluating, and generalizing humanoid robot policies.

What This Paper Is About

Humanoid robots need reference motion data that is both broad (covering many whole-body behaviors) and physically exact, but existing sources fail one of those two tests: internet video is broad but lacks precise physical state, while laboratory motion capture is precise but narrow in behavioral coverage. HiPHI addresses this by building a motion capture dataset whose coverage is deliberately organized using FrameNet, a linguistic framework that treats word meanings as event types, so that motion collection becomes a systematic expansion of motion units rather than an ad hoc list of hand-written scripts. The authors capture 617.5 hours of high-fidelity whole-body motion and real-object interaction and then evaluate it for motion-space coverage, data quality, interaction consistency, and downstream humanoid policy learning.

Key Contributions

1. A FrameNet-guided motion-space construction pipeline. Frames and lexical units (LUs) relevant to embodied intelligence are selected, and each Frame–LU pair becomes a seed expanded along factors such as direction, speed, amplitude, posture, body-part involvement, support relation, and object/contact conditions. The authors state this makes HiPHI, to their knowledge, the first MoCap dataset for robot learning to prospectively adopt such a linguistic scaffold.

2. A 617.5-hour, high-precision dataset. HiPHI contains 371.8 hours of body-only motion and 245.7 hours of human–object interaction, with synchronized object trajectories and meshes, Frame–LU indexing, and natural-language descriptions, captured from 132 performers at 90 Hz with sub-millimeter spatial tracking accuracy.

3. A comprehensive evaluation protocol for humanoid robot learning. The benchmark measures motion coverage, data quality, interaction consistency, motion tracking, and real-robot deployment, rather than only dataset statistics.

Main Findings

• Scale and composition: HiPHI provides 617.5 hours of optical motion capture in BVH format, corresponding to approximately 200.1 million frames at 90 Hz. This results from applying left-right mirroring to 308.7 hours of original captured motion, following the convention used by BONES-SEED. The human-object interaction share is 39.8%.

• Semantic indexing: The release contains 214 Frame–LU labels across 22 FrameNet frames. The top 10 Frame-LUs account for 20.4% of duration, the top 20 for 31.8%, and the top 50 for 53.7%, leaving 46.3% in the long tail. The median number of actors per Frame–LU is 24, and 154 Frame-LUs are performed by at least 10 actors.

• Object-interaction subset: 245.7 hours of strict object-interaction motion, spanning 90 Frame-LUs across 15 FrameNet frames, with 40 real-world objects from 12 categories and masses from 0.45 to 6.25 kg.

• Broader kinematic coverage: Under a shared unsupervised body-motion encoder and a common t-SNE projection, HiPHI covers more grid cells than the closest baseline BONES-SEED (1620 vs. 1438), with larger effective occupancy (1443 vs. 1114) and higher long-tail share (14.1% vs. 10.7%). The authors report that HiPHI retains a positive occupied-cell margin over the strongest baseline across multiple random seeds, t-SNE perplexities, and grid resolutions.

• Better motion precision: Among datasets sharing the same floor-plane convention, HiPHI achieves the best value on every reported body-motion metric: jerk 173.9 m/s³, acceleration 10.7 m/s², below-ground depth 8 mm, unsupported-floating share 0.015%, and support-point drift 64 mm/s. For comparison, AMASS reports 529.0, 14.9, 111, 0.76, and 87; BONES-SEED reports 294.2, 11.3, 18, 1.02, and 86; LAFAN1 reports 383.1, 23.4, 29, 7.22, and 157. Motion-X++ is excluded from the three floor-related metrics because it lacks a comparable absolute ground convention.

• Stronger interaction geometry: HiPHI reaches 98.1% non-conflict fraction and 95.7% near-surface grounding versus HIMO at 97.6% and 79.0%, OMOMO at 90.6% and 50.4%, and HUMOTO at 99.9% and 82.9%. HiPHI's object-interaction duration (245.7 h) is much longer than HIMO's 21.6 object-track hours and OMOMO's 9.8 h.

• Better humanoid tracking: With all motions retargeted to the Unitree G1 and trained with the same DeepMimic imitation pipeline, HiPHI achieves the highest success rates and fastest convergence in both matched 3-hour and 20-hour settings, with the advantage consistent across five independent training runs.

• Data scaling: Progressively increasing unmirrored HiPHI training data from 3 to 300 hours consistently reduces cross-dataset MPJPE when evaluating on AMASS, BONES-SEED, Motion-X++, and LAFAN1.

• Motion-with-object tracking: HiPHI achieves the best overall body tracking across most categories (for example, kick MPJPE 26.31 mm and velocity error 3.68 mm/frame, carry 50.33 and 4.14, lean 30.76 and 2.12), while on push its MPJPE of 99.20 mm is higher than OMOMO's 66.09 mm. For object tracking, HiPHI obtains the best position and orientation accuracy on kick and push, and the best orientation accuracy on carry. HUMOTO lacks push motions and OMOMO lacks lean motions, marked N/A.

• Real-robot transfer: Policies trained with HiPHI were deployed on the Unitree G1 and performed running, sitting, crawling, carrying a box, flipping, and pulling a suitcase.

Methodology in Plain English

The authors treat data collection as a design problem over "motion space" rather than as script writing. They start from FrameNet, which organizes meanings into event types (frames) and the specific word senses that evoke them (lexical units). For example, walk, jog, and run fall under the Self_motion frame, and a Frame–LU pair pins down one meaning precisely instead of treating every use of a word the same way. They select embodied-intelligence-relevant frames and LUs, then expand each one along controllable factors such as path, direction, speed, rhythm, amplitude, posture, support relation, and object/contact conditions. These expansions become short, repeatable instructions that human performers execute in a large optical motion capture system. Objects are captured alongside the human body so that objects' geometry, load, friction, and trajectory are recorded as part of the motion record.

For evaluation, the researchers compare HiPHI against existing datasets along separate axes. Motion-space coverage is measured by encoding all datasets with the same unsupervised body-motion encoder and projecting the codes into one shared t-SNE space, then counting how many grid cells each dataset occupies and how uniformly. Data quality is measured with five body-motion quantities and two object-interaction quantities. Downstream usefulness is tested by retargeting motions to the Unitree G1 and training policies with the same imitation-learning pipeline, using matched data budgets (3 hours, and 20 hours for larger datasets) so that differences reflect data usability rather than scale. A separate scaling experiment increases HiPHI training data from 3 to 300 hours, and a final deployment step tests trained policies on the physical robot.

Why This Matters

Impact on research. The paper reframes dataset construction as a coverage problem with a testable taxonomy, rather than an open-ended accumulation of scripts. It provides a shared evaluation protocol combining coverage statistics, physical quality metrics, interaction-consistency metrics, and closed-loop policy performance. The authors also note that similar extensions are applicable to motion prior models in computer graphics.

Real-world applications:

• Training humanoid controllers for whole-body behaviors such as locomotion, posture transitions, sitting, and crawling.
• Learning object-constrained skills such as pushing, pulling, carrying, leaning, and supporting loads, where object mass and geometry change the required control strategy.
• Sim-to-real deployment of imitation-trained policies on hardware, demonstrated here on the Unitree G1.
• Providing motion priors and reference data for animation and character motion generation in graphics.

Industry relevance. The dataset targets the practical bottleneck in humanoid development: reference motion that is simultaneously large and physically faithful. Because object masses, trajectories, and meshes are recorded, the data supports skill training where contact and load matter, and the paper's matched-budget protocol offers a way for teams to compare data sources on policy performance rather than on headline hours alone.

Future Directions

• Multi-person interaction. The authors state HiPHI currently captures only single-person motion; multi-person interaction and human-human contact remain outside its scope and require complementary efforts.
• Contact forces and tactile signals. HiPHI focuses on kinematics and does not directly measure contact forces or tactile signals, which would be needed for richer physical grounding.
• In-the-wild collection. The dataset is collected in a studio; the authors identify capturing accurate motion and object interaction with ego-vision in the wild as a promising future direction.
• Extending the semantic scaffold. Because FrameNet is used as a semantic scaffold rather than a closed taxonomy, the frame and lexical-unit selection, expansion factors, and coverage statistics could be extended or adapted to other data-collection efforts.

Target Audience

Researchers and engineers working on humanoid robot learning, reinforcement-learning-based motion imitation, and sim-to-real transfer; dataset and benchmark builders in robotics and embodied AI; and graphics researchers who need high-fidelity human motion and human-object interaction data for motion synthesis and character animation. Readers with a background in machine learning or robotics but limited motion capture experience will find the pipeline description and evaluation protocol accessible, since the paper explains the FrameNet scaffolding and each quality metric in concrete terms.

Authors’ abstract

Humanoid intelligence requires learning over an extremely diverse space of whole-body motions and physically grounded interactions. However, existing embodied datasets remain fundamentally limited: internet-scale video data lack precise physical states and interaction grounding, while laboratory motion datasets provide high fidelity but only narrow behavioral coverage. This mismatch creates a critical bottleneck for scalable humanoid policy learning. We present HiPHI, a 600+ hour scale high-fidelity whole-body human motion dataset designed to systematically maximize coverage of the human motion and interaction manifold. HiPHI is theoretically guided by FrameNet, a linguistic framework organizing human primitives. Created using an optical motion capture pipeline, HiPHI provides sub-millimeter spatial marker tracking accuracy for full-body human motion and mesh-level object trajectories. We further introduce a benchmark suite evaluating motion-space diversity, interaction grounding, object consistency, and physical AI applications. Our analyses demonstrate that HiPHI significantly expands motion coverage compared to existing motion datasets while maintaining high-fidelity interaction quality, and establishes a scalable data foundation for training, evaluating, and generalizing humanoid policies in real-world embodied tasks, where similar extensions are also applicable to motion prior models in computer graphics.