UnrealPose: Leveraging Game Engine Kinematics for Large-Scale Synthetic Human Pose Data

AI summary

Overview

Research area: Computer vision — 3D human pose estimation and synthetic dataset generation, specifically using the Unreal Engine 5 game engine to produce labeled training data.

Technical level: Intermediate. The paper is readable without deep graphics expertise, but familiarity with pose estimation tasks (2D keypoints, 2D-to-3D lifting, human mesh recovery) helps.

Scope in one sentence: The paper introduces UnrealPose-Gen, an Unreal Engine 5 pipeline that renders annotated human pose frames from game-engine skeletons, and uses it to build UnrealPose-1M, an approximately one million frame dataset evaluated on four pose-related tasks.

What This Paper Is About

Accurate 3D human pose data is expensive to capture in a studio, and in-the-wild image sets typically lack reliable 3D ground truth. Many researchers instead generate pseudo-3D labels from 2D data using lifting networks or SMPL-family body models, but these approaches inherit biases from their source datasets and model assumptions — for example, CAESAR scan demographics or errors introduced when joints are regressed from a fitted mesh rather than taken from anatomical pivot points.

The paper's goal is to sidestep those biases by labeling the actual kinematic joints of Unreal Engine 5 MetaHuman skeletons, rather than joints derived from a mesh, and to package that capability into a reusable generator plus a large released dataset.

Key Contributions

1. UnrealPose-Gen, a UE5/Movie Render Queue pipeline that outputs, per frame: 3D rig joints in world and camera coordinates, 2D projections and COCO-style keypoints with occlusion and joint-visibility flags, person bounding boxes and instance segmentation masks, and camera intrinsics and extrinsics. The camera-centric design also supports online rendering during gameplay, not just offline rendering.

2. UnrealPose-1M, a corpus of approximately one million annotated frames: five scripted "coherent" sequences spanning five scenes, approximately 40 actions, and five subjects, plus three randomized sequences across three scenes, approximately 100 actions, and five subjects.

3. SMPL-independent labeling: annotations come from engine-native skeletal pivots that drive animation, so any permitted UE skeleton — marketplace packs, retargeted motion capture, even SMPL motions retargeted to UE rigs — can be rendered with annotations. The pipeline tracks up to 255 character assets.

4. A four-task fidelity evaluation covering image-to-3D pose, 2D keypoint detection, 2D-to-3D lifting, and person detection/segmentation, using pretrained models as independent evaluators. Both the pipeline and dataset are publicly released.

Main Findings

• Dataset composition: UnrealPose-1M contains approximately 800,000 frames from coherent sequences across five scenes, approximately 170,000 frames from randomized sequences across three scenes, and approximately 115,000 multi-person frames across two scenes. The split is 75/20/5 train/validation/test, where consecutive kept frames are at least 100mm apart in summed joint Euclidean distance in camera coordinates.

• Per-frame annotations: 17 COCO-Pose format 2D keypoints with visibility flags, 16 common skeletal joints projected to 2D with visibility flags, 16 common 3D joints in world and camera coordinates, and per-person bounding boxes plus segmentation masks with unique IDs maintained across frames.

• Camera diversity: FOV was varied from approximately 30° to approximately 90°, with heights from ground level to overhead, and each coherent sequence uses 15–20 static camera positions. Distances to the subject vary within each static-camera sequence, producing close and far framings.

• 2D keypoint detection: On a test set of 38,050 images containing 42,528 annotated person instances, top-down HRNet-W48 reached AP 0.883, AP50 0.990, AP75 0.980, and AR 0.896. Bottom-up DEKR-HRNet-W32 reached AP 0.802, AP50 0.977, AP75 0.923, and AR 0.831, evaluated at image level on the same 38,050-frame test set. The paper notes these high scores reflect annotation consistency and metric compatibility rather than task difficulty, and that they are non-saturated.

• 2D-to-3D lifting: PoseAug, trained on Human3.6M and never fine-tuned on the synthetic data, achieved MPJPE of 61.81 mm and PA-MPJPE of 57.28 mm, evaluated on approximately 55,000 sampled synthetic frames with root-joint translation alignment.

• Per-joint error pattern for lifting: Torso joints such as the neck, spine, and hips showed lower errors, while distal joints including elbows, wrists, knees, and ankles showed higher errors. The pelvis joint showed the highest raw error because it serves as the alignment root.

• Image-to-3D regression: MeTRAbs, trained on multiple real-world datasets and not fine-tuned on synthetic data, reconstructed coherent 3D skeletons. Table 1 reports MPJPE of 104.16 mm and PA-MPJPE of 111.41 mm, while the Section 4.3 text reports MPJPE of 99.17 mm and PA-MPJPE of 100.51 mm on approximately 38,000 sampled images — the two reported figures differ within the paper.

• Instance segmentation: Mask2Former for panoptic segmentation with a Swin-L backbone pretrained on COCO achieved an average IoU of 0.89 on the test set, and diverse scene elements (sky, vases, trees) were also correctly labeled.

• Evaluation constraints: Full training on UnrealPose-1M was not feasible under the authors' compute constraints, so the study is an evaluation using pretrained models rather than new model training.

Methodology in Plain English

The authors build their generator inside Unreal Engine 5's camera system. A user selects which characters to track (up to 255), and the system records, for every rendered frame, the character's skeletal joint positions in world space, then transforms them into camera space. Those 3D points are projected into 2D image coordinates, and a second set of standard COCO-Pose keypoints is also exported. To decide whether a joint is visible, the system traces a line from the camera to the joint's world point — if something blocks the line, the joint is flagged as occluded. Bounding boxes and segmentation masks are occlusion-aware: masks are cut at the occluding boundary, and boxes tightly enclose only the visible portion of each person.

To build the dataset, the authors wrote a movement driver with two modes. In scripted mode, characters walk between defined markers using locomotion animations and use idle animations when stationary, producing temporally coherent sequences for video-based methods. In random mode, characters pick random locations within a bounded area and play randomly selected animations from a directory, maximizing pose, viewpoint, and action diversity for single-frame methods. Cameras are static, so intrinsics and extrinsics are exported once per camera rather than per frame.

Two filters trim the output: frames where a tracked person has keypoints projecting outside the image bounds are discarded, and frames that are not sufficiently different in joint Euclidean distance from the previous frame are discarded as near-duplicates.

For evaluation, the authors use pretrained, publicly available models as independent judges — none were fine-tuned on the synthetic data — and compare their predictions against the engine's ground-truth annotations under root-joint translation alignment, which focuses the error measurement on relative body geometry rather than global position.

Why This Matters

The work argues that game animation libraries already contain the interaction-rich motions — multi-person fighting, conversations, object manipulation — that the computer vision community has struggled to synthesize, and that even state-of-the-art synthetic datasets acknowledge generating realistic synthetic human-object and human-human interaction "remains an open research problem." Tapping those assets changes where pose training data can come from, and labeling kinematic pivots rather than regressed mesh joints removes one source of systematic label bias.

Real-world applications:

• Pose estimation in crowded, occluded settings, since the dataset provides occlusion-aware boxes and masks plus per-joint visibility flags, enabling training on crowded scenes where standard datasets are weak.
• Domain-specific data generation from existing UE5 games, using the pipeline's real-time online rendering to produce training data tailored to a particular application without building a capture rig.
• Sports, combat, and collaborative-task analysis, because the animation libraries cover motions that are difficult or unsafe to motion-capture.
• Unconventional viewpoint coverage, since the varied FOV (approximately 30° to approximately 90°) and ground-level to overhead camera heights expose models to views rare in standard benchmarks, which tend to use canonical third-person views.

Industry relevance: Game developers already own the animation content the pipeline consumes, and the authors frame the generator as a way to turn that asset library into a data resource. Anyone with a UE5 project and a valid skeleton — marketplace assets, custom characters, retargeted motion capture — can render fully annotated frames, lowering the barrier to producing custom pose datasets without a studio.

Future Directions

• Packaging as a UE5 plugin. UnrealPose-Gen currently requires manual integration into UE5 projects; a plugin would remove the need to modify a project or understand the codebase.
• Scaling characters, skeletons, and animations. The five MetaHumans represent a tiny fraction of what the MetaHuman creator can produce, and the pipeline also works with non-MetaHuman UE meshes, amputees, and non-human entities. The UE marketplace contains thousands of character models and animations, and the scaling laws of synthetic pose data remain an open question.
• Moving cameras and dynamic intrinsics. The current implementation uses static cameras and exports intrinsics/extrinsics once per camera; supporting moving cameras would simulate more realistic footage and possibly reduce the number of static camera positions needed.
• Training rather than only evaluating. Because compute constraints prevented training pose models on UnrealPose-1M, the authors plan training from scratch and fine-tuning to test synthetic-to-real transfer against other synthetic datasets. They also have not yet validated online rendering in production game environments.

Target Audience

Researchers and engineers working on 3D human pose estimation, 2D-to-3D lifting, human mesh recovery, and synthetic data generation; game developers and UE5 users interested in harvesting existing animation assets for machine learning; and teams that need custom, occlusion-aware pose annotations but cannot afford studio capture. It is also useful for readers evaluating the tradeoffs between SMPL-centric mesh supervision and kinematic-joint supervision.

Authors’ abstract

Diverse, accurately labeled 3D human pose data is expensive and studio-bound, while in-the-wild datasets lack known ground truth. We introduce UnrealPose-Gen, an Unreal Engine 5 pipeline built on Movie Render Queue for high-quality offline rendering. Our generated frames include: (i) 3D joints in world and camera coordinates, (ii) 2D projections and COCO-style keypoints with occlusion and joint-visibility flags, (iii) person bounding boxes, and (iv) camera intrinsics and extrinsics. We use UnrealPose-Gen to present UnrealPose-1M, an approximately one million frame corpus comprising eight sequences: five scripted "coherent" sequences spanning five scenes, approximately 40 actions, and five subjects; and three randomized sequences across three scenes, approximately 100 actions, and five subjects, all captured from diverse camera trajectories for broad viewpoint coverage. As a fidelity check, we report real-to-synthetic results on four tasks: image-to-3D pose, 2D keypoint detection, 2D-to-3D lifting, and person detection/segmentation. Though time and resources constrain us from an unlimited dataset, we release the UnrealPose-1M dataset, as well as the UnrealPose-Gen pipeline to support third-party generation of human pose data.