Sketch2PoseNet: Efficient and Generalized Sketch to 3D Human Pose Prediction

AI summary

Overview

Research area: Computer vision and computer graphics — specifically sketch-based 3D human pose and shape estimation (character posing / motion capture).

Technical level: Advanced. The paper combines latent diffusion models, ControlNet conditioning, SMPL body modelling, and feed-forward pose regression; it assumes familiarity with diffusion priors, heatmap-based 2D pose estimation, and SMPL parameter regression.

Scope: The paper introduces a synthetic 120k-pair sketch–3D pose dataset (SKEP-120K) and a feed-forward network that recovers SMPL-based 3D human poses from stylistically diverse sketches roughly 500 times faster than the prior state-of-the-art sketch-to-pose method.

What This Paper Is About

Estimating a 3D human pose from a drawing is hard because sketches are abstract: they ignore real body proportions, use inconsistent perspective, and come in wildly different styles (cartoons, ink paintings, stick figures, and so on). Prior work, chiefly Sketch2Pose, had to rely on iterative optimization with hand-crafted rules because no large paired sketch-to-3D dataset existed — making it slow and narrow in the styles it could handle.

The goal of this work is to build a large-scale sketch-to-3D pose dataset and train an end-to-end neural network on it, so that pose prediction from sketches becomes both accurate across styles and fast enough for practical, near-real-time use.

Key Contributions

1. A "learn from synthesis" pipeline for sketch–pose data. Using pose perturbations (random bone-length biases) to mimic the disproportionate figures in drawings, the authors synthesize SKEP-120K, a dataset of 120k sketch–3D pose annotation pairs spanning six sketch styles, each contributing roughly 20,000 images.

2. A feed-forward sketch-to-pose network. The method replaces optimization with a three-module architecture — a 2D guidance extractor, a sketch feature extractor built on a pre-trained denoising U-Net, and an SMPL mesh regressor — achieving an over 500× speedup compared with the prior state-of-the-art sketch-to-pose estimator.

3. A loss design tailored to human figure drawing. Three drawing-specific principles — joint angle, foreshortening, and self-contacts — are encoded as a skeleton parallelism loss, a foreshortening loss, and SMPL pose/shape parameter losses, replacing position-based reprojection losses.

4. State-of-the-art accuracy plus broad generalization. The method reports the best overall metrics on both the artist-designed dataset from Sketch2Pose and the SKEP-120K validation set, and holds up across sketch styles where generic photo-based pose estimators degrade.

Main Findings

• Best overall accuracy on the artist-designed dataset. On Expert1, the method reports MPVE 103.1 mm, MPJPE 95.7 mm, PA-MPJPE 77.4 mm; on Expert2, MPVE 146.5 mm, MPJPE 131.5 mm, PA-MPJPE 84.3 mm. Sketch2Pose reports 103.8 / 101.4 / 78.1 and 145.5 / 135.9 / 86.8 respectively. The authors state gains over Sketch2Pose are modest on charcoal sketches and that they perform slightly worse on one expert-based metric (Expert2 MPVE).

• Large margin on the SKEP-120K validation set. The method reports MPVE 106.7 mm, MPJPE 87.7 mm, PA-MPJPE 72.6 mm, compared with Sketch2Pose at 152.1 / 125.9 / 100.3 and the retrained DPMesh at 122.6 / 97.3 / 80.6.

• Substantial runtime advantage. Total per-input runtime is 0.12 s for the proposed method versus 67.57 s for Sketch2Pose (stages: 0.04 s / 0.05 s / 0.03 s versus 4.75 s / 32.98 s / 30.15 s). This is described as an over 500× speedup, and the paper also compares against photo-based methods such as HMR2.0 (0.55 s), MotionBERT (0.19 s), DPMesh (0.14 s), HyBrIK (0.17 s), EFT (3.57 s), CLIFF (3.85 s), and PyMAF (0.10 s).

• Every ablated component matters. Removing the parallelism loss gives MPVE 169.8 / MPJPE 165.7 / PA-MPJPE 102.5; removing the foreshortening loss gives 117.7 / 110.2 / 84.4; removing the pose loss gives 121.4 / 117.6 / 86.7; removing all three gives 113.7 / 107.3 / 85.6; removing the cross-attention maps A_i gives 104.9 / 97.1 / 79.6; removing the 2D joint guidance gives 117.4 / 113.9 / 89.6; skipping data curation gives 104.8 / 99.0 / 82.7 — all versus the full model at 103.1 / 95.7 / 77.4 on Expert 1.

• Training data quality matters. Models trained on the manually curated set outperform those trained on the pre-curation set. About 10% of sketches per style were filtered out by experienced 3D modelers for cluttered backgrounds, pose inconsistencies, or extreme distortion.

• Robustness to occlusion and overlapping strokes. Qualitative results show plausible poses when an arm is fully occluded and when character lines overlap with a handheld object.

• Video and retargeting capability. The inference speed allows frame-by-frame prediction on continuous line animations, and predicted poses are retargeted onto custom characters.

• Not reported: the main text does not give a numeric user-study result — it states that a subjective user study is provided in the supplementary material. Detailed training schedules are also deferred to the supplementary material.

Methodology in Plain English

The authors start from the observation that nobody has enough real sketches paired with true 3D poses to train a deep model. So they manufacture that data. They use VPoser to generate diverse, plausible SMPL body poses, then deliberately distort the bone lengths when projecting those poses to 2D — imitating how artists exaggerate proportions. Existing sketch datasets (Sketch2Pose, Human-Art, Amateur Drawing) are unified to a common 16-joint skeleton based on MSCOCO with two added toe joints, then grouped into six drawing styles. BLIP2 writes appearance descriptions of sketches and motion descriptions of the rendered bodies; those descriptions plus a style label become the text prompt. A ControlNet-style text-conditioned image generator is trained on all six styles jointly and used to synthesize the sketches. Finally, human 3D modelers review the output and remove bad samples.

The prediction network takes a sketch, resizes and pads it to 256×192, and uses a fine-tuned YOLOX detector plus a fine-tuned ViTPose model to find 2D joints and convert them into heatmaps. These 2D cues are injected into a pre-trained denoising U-Net — heatmaps concatenated with the image latents, and 2D joint positions passed through a two-layer MLP into a 768-dimensional token space to replace the usual text embedding. The U-Net then runs a single forward pass, and multi-scale features are pulled from the last layer of each output block and concatenated with cross-attention maps, which act as occlusion-aware cues. A fusion transformer lifts these features to 3D and regresses SMPL parameters, with the shape and camera parameters predicted by linear layers and the pose decoded via a VQVAE decoder pre-trained on AMASS.

Training uses a combined objective: a skeleton parallelism loss (L_parallel) enforcing that projected 3D bones align with the drawn 2D bones, a foreshortening loss (L_f) comparing the ratio of 3D bone length to its 2D projection, and L1 losses on SMPL pose and shape parameters. Weights λ1, λ2, λ3, λ4 are set to 3, 3, 2, 1. All training and experiments ran on 4 Nvidia A6000 GPUs; ControlNet condition maps and sketches were padded to 512×512.

Why This Matters

Impact on research. The paper reframes sketch-to-pose as a data problem rather than an optimization problem. It shows that a synthetic, style-diverse dataset with deliberate proportion perturbations can carry an end-to-end model past a heavily engineered optimization baseline, and it demonstrates a concrete way of reusing diffusion priors for a non-photorealistic input domain.

Real-world applications:

• Character animation and film pre-production — artists sketch a pose and get a rigged 3D body back immediately, without waiting for iterative solvers.
• Rapid prototyping in game and VR content pipelines — near-real-time inference (0.12 s per input) supports interactive posing tools.
• Frame-by-frame pose transfer from line animation — the speed enables processing continuous sketches as a sequence rather than isolated images.
• Motion capture and retargeting — predicted SMPL poses can be transferred onto different characters, as shown in the paper's figures.

Industry relevance. The headline numbers are practical ones: 0.12 s versus 67.57 s turns an offline, expert-driven workflow into something approaching interactive. The reported runtime is competitive with photo-based pose estimators (DPMesh at 0.14 s, HyBrIK at 0.17 s, PyMAF at 0.10 s), which matters for studios that want one pipeline serving both photographs and concept art.

Future Directions

1. Better handling of extreme cases. The paper includes a "Failure" figure but the truncated content does not describe its contents; understanding and addressing these failure modes is a natural next step.

2. More rigorous subjective evaluation. The main text defers the user study to the supplementary material, leaving room for a larger, more systematically reported study of whether predictions match artists' intent.

3. Scaling and diversifying sketch styles. The six current styles (cartoons, oil paintings, ink paintings, charcoal sketches, stick figures, kids' drawings) could be extended, and the synthesis pipeline could be re-run for new styles.

4. Beyond static poses. The paper notes that fast inference enables application to video, and demonstrates frame-by-frame results on continuous line animations; extending this to temporally coherent motion recovery is an open direction.

Target Audience

Researchers and practitioners in computer graphics and computer vision working on character posing, sketch-based modeling, and human mesh recovery; technical artists and animation pipeline engineers who need fast sketch-to-3D tools; and anyone interested in how diffusion model priors can be adapted to non-photorealistic input domains.

Authors’ abstract

3D human pose estimation from sketches has broad applications in computer animation and film production. Unlike traditional human pose estimation, this task presents unique challenges due to the abstract and disproportionate nature of sketches. Previous sketch-to-pose methods, constrained by the lack of large-scale sketch-3D pose annotations, primarily relied on optimization with heuristic rules-an approach that is both time-consuming and limited in generalizability. To address these challenges, we propose a novel approach leveraging a "learn from synthesis" strategy. First, a diffusion model is trained to synthesize sketch images from 2D poses projected from 3D human poses, mimicking disproportionate human structures in sketches. This process enables the creation of a synthetic dataset, SKEP-120K, consisting of 120k accurate sketch-3D pose annotation pairs across various sketch styles. Building on this synthetic dataset, we introduce an end-to-end data-driven framework for estimating human poses and shapes from diverse sketch styles. Our framework combines existing 2D pose detectors and generative diffusion priors for sketch feature extraction with a feed-forward neural network for efficient 2D pose estimation. Multiple heuristic loss functions are incorporated to guarantee geometric coherence between the derived 3D poses and the detected 2D poses while preserving accurate self-contacts. Qualitative, quantitative, and subjective evaluations collectively show that our model substantially surpasses previous ones in both estimation accuracy and speed for sketch-to-pose tasks.