Stroke3D: Lifting 2D strokes into rigged 3D model via latent diffusion models

AI summary

Overview

• Research area: 3D generative modeling and computer graphics/vision — specifically automatic rigging, skeleton generation, and skeleton-conditioned mesh synthesis.
• Technical level: Advanced. The paper assumes familiarity with latent diffusion models (DiT), graph neural networks, variational autoencoders, CLIP text conditioning, and Direct Preference Optimization (DPO).
• Scope: The paper proposes Stroke3D, a two-stage latent-diffusion framework that turns user-drawn 2D strokes plus a text prompt into an animatable, rigged 3D mesh.

What This Paper Is About

Existing 3D generation methods mostly produce static geometry that cannot be animated, and existing rigging methods generate skeletons from meshes without letting a user control where joints go, so skeletons appear in unnecessary places and are missing where they are needed. Stroke3D instead starts from a skeleton: the user draws 2D strokes on a canvas tool and writes a text prompt, and the system generates a 3D skeleton, then a textured mesh conditioned on that skeleton. The authors state this is the first work to generate rigged 3D meshes conditioned on user-drawn 2D strokes.

Key Contributions

1. Stroke-first rigged generation. The authors present Stroke3D, described as the first framework to generate rigged 3D meshes directly from user-drawn 2D strokes and text prompts, via a skeleton-first pipeline built on a latent graph diffusion model.
2. A new skeleton representation pipeline. They introduce a Skeletal Graph VAE (Sk-VAE) that encodes skeleton graph structure into a latent space, and a Skeletal Graph Diffusion Transformer (Sk-DiT) that generates skeletal embeddings conditioned on text and 2D strokes.
3. The TextuRig dataset. A curated dataset of textured, rigged 3D meshes with captions, sourced from Objaverse-XL (built on the rigged subset filtered by UniRig), used to augment training data for the skeleton-to-mesh model SKDream.
4. SKA-DPO preference optimization. A tailored DPO strategy guided by the skeleton-mesh alignment (SKA) score to improve geometric fidelity and skeleton-mesh correspondence.

Main Findings

• Skeleton generation accuracy. On the MagicArticulate test set, Stroke3D reports the lowest Chamfer Distance on most metrics, with CD-J2J of 0.048, CD-J2B of 0.039, and CD-B2B of 0.034 across all categories. For comparison, the table lists RigNet at 0.078/0.066/0.065, SKDream at 0.111/0.092/0.083, MagicArticulate at 0.052/0.041/0.034, and UniRig at 0.063/0.051/0.041 on the same three "All" metrics.
• Balanced across categories. MagicArticulate and UniRig are described as competitive on characters but less consistent on animals and plants; Stroke3D reports the lowest plant scores (CD-J2J 0.040, CD-J2B 0.027, CD-B2B 0.021) and balanced improvements across all three categories.
• TextuRig improves mesh alignment. Adding TextuRig to the SKDream baseline raises the SKA score from 80.43 to 82.37 (Mean Inst.) and from 74.38 to 76.84 (Mean Class) — a 1.9 and 2.4 improvement, respectively, as reported by the authors.
• SKA-DPO adds a further gain. Applying SKA-DPO alone yields SKA scores of 85.57 Mean Inst. and 81.12 Mean Class; combining both TextuRig and SKA-DPO gives the best results at 87.83 Mean Inst. and 84.36 Mean Class. The introduction states the method improves the Mean Inst. SKA score by nearly 10 points over the SKDream baseline (Table 2 lists 87.83 versus 80.43).
• Subclass behavior. On the animal subclasses, the full method scores 93.75 (Apodes), 70.63 (Bipeds), and 93.33 (Wings). The Wings score is slightly lower than SKDream's 94.47, while Bipeds improves substantially from SKDream's 53.53.
• Preference margin matters. In the SKA-DPO margin ablation, 0.05 gives 86.95/83.03, 0.10 gives 87.83/84.36, 0.15 gives 87.84/84.21, and 0.20 gives 86.96/83.12 (Mean Inst. / Mean Class). The authors adopt 0.10 as an optimal trade-off.
• Structural conditioning speeds learning. Training from scratch on datasets of one or five samples, models conditioned on the joint XY coordinates (Jxy) reach lower loss far more quickly than models without it; the authors note convergence struggles at large-scale training without this condition.
• Robustness to sparse input. The model maintains high stability and low CD scores when a small number of joints (fewer than 5) are dropped, suggesting resilience to minor occlusions or incomplete sketches.
• Animation stability. After binding the generated meshes to skeletons with auto-skinning tools, the meshes maintain structural integrity without collapsing.

Methodology in Plain English

The system is split into two stages.

Stage one — generating a skeleton. The user's strokes are captured by a canvas tool that records clicked 2D joint positions and the lines connecting them, producing a graph that mirrors how skeletons are defined in software like Blender (joints plus bones). This graph is used as an explicit structural condition, since text alone cannot determine how many joints an object needs or how they connect. A variational autoencoder with graph layers (GCN and TransformerConv) compresses 3D skeletons into a compact latent representation with a slight KL penalty. A DiT-style diffusion transformer then generates new latent skeleton embeddings; it swaps standard self-attention for TransformerConv so attention only flows along graph edges, adds cross-attention to CLIP text embeddings for semantics, and concatenates stroke features with the noisy latent for structure. To train this without requiring real hand-drawn data, the authors simulate strokes by taking 2D projections of ground-truth 3D skeletons and perturbing them. The VAE decoder then reconstructs the final 3D skeleton.

Data preparation. Because skeletons rendered alone are abstract (a mushroom skeleton looks like a stick), the authors render the mesh and skeleton together in orthogonal projections and feed those views to a vision-language model to generate rich captions. For TextuRig, they start from the UniRig-filtered rigged subset of Objaverse-XL, re-process raw assets to preserve materials, filter for models that actually have texture maps or vertex colors, and re-caption with Gemini. The appendix describes aligning all training models to a canonical orientation (facing +Z, head +Y, left +X) using joint-name heuristics for human-like subjects, leg symmetry for structural cases, and principal-direction/root-joint heuristics for animals and plants.

Stage two — generating the mesh. SKDream is fine-tuned with the augmented data (SKDream's roughly 24,000 samples plus 6,800 TextuRig annotations), using TextuRig captions as semantic prompts. Then SKA-DPO is applied: for each skeleton-caption pair, two multi-view candidates are generated with different noise seeds, a fine-tuned Dinov2 model from SKDream computes the SKA Score for each, and the higher-scoring one becomes the "winner" and the other the "loser." The model is then trained with a DiffusionDPO objective to favor winners over losers relative to a reference model.

Why This Matters

• Research impact: The paper argues that decoupling skeleton creation from geometry generation — the standard mesh-to-skeleton paradigm — limits user control, and that conditioning on 2D strokes supplies an explicit structural prior that is missing from text-only or mesh-only pipelines. It also contributes TextuRig as a resource addressing the shortage of textured, rigged, captioned 3D data.
• Real-world applications:
  • Rapid prototyping of animatable characters, animals, and plants for games and animation without expert rigging skills.
  • AR/VR content pipelines that need ready-to-animate assets produced quickly from sketches.
  • Robotics simulation, which the authors list among the domains that rely on rigged 3D assets.
  • Film and animation pre-production, where artists sketch a pose and get a rigged model back.
  • Structural editing workflows shown in the paper: regenerating from different viewpoints, adding strokes, or moving joint positions.
• Industry relevance: The output is compatible with mature automatic skinning tools such as those in Blender, meaning generated skeleton-mesh pairs can be dropped into existing professional pipelines. The stated motivation is lowering the steep learning curve of professional 3D software for non-experts.

Future Directions

• The Wings subclass score of 93.33 is slightly below SKDream's 94.47, and Bipeds remains the weakest subclass at 70.63, suggesting room for improvement on particular anatomy types.
• The paper notes that texture retrieval from the heterogeneous MagicArticulate dataset was difficult, forcing the mesh stage to use the SKDream dataset instead; extending texture-aware training data is an open direction.
• Sensitivity analysis covers dropping fewer than 5 joints; behavior under heavier occlusion or much rougher user sketches is not reported.
• The appendix notes that canonical orientation was enforced through heuristics (joint names, leg symmetry, principal directions). Automating or generalizing alignment to categories where these heuristics fail is left open.

Target Audience

Researchers and practitioners in 3D generative modeling, computer graphics, and character animation who work on rigging, skeleton generation, or diffusion-based 3D asset creation. It is also relevant to technical artists and pipeline engineers interested in sketch-driven content creation, though the paper's reliance on graph VAEs, DiT architectures, and preference optimization makes it an advanced read. Beginners can follow the high-level workflow but will need background in diffusion models and graph neural networks to evaluate the technical details.

Authors’ abstract

Rigged 3D assets are fundamental to 3D deformation and animation. However, existing 3D generation methods face challenges in generating animatable geometry, while rigging techniques lack fine-grained structural control over skeleton creation. To address these limitations, we introduce Stroke3D, a novel framework that directly generates rigged meshes from user inputs: 2D drawn strokes and a descriptive text prompt. Our approach pioneers a two-stage pipeline that separates the generation into: 1) Controllable Skeleton Generation, we employ the Skeletal Graph VAE (Sk-VAE) to encode the skeleton's graph structure into a latent space, where the Skeletal Graph DiT (Sk-DiT) generates a skeletal embedding. The generation process is conditioned on both the text for semantics and the 2D strokes for explicit structural control, with the VAE's decoder reconstructing the final high-quality 3D skeleton; and 2) Enhanced Mesh Synthesis via TextuRig and SKA-DPO, where we then synthesize a textured mesh conditioned on the generated skeleton. For this stage, we first enhance an existing skeleton-to-mesh model by augmenting its training data with TextuRig: a dataset of textured and rigged meshes with captions, curated from Objaverse-XL. Additionally, we employ a preference optimization strategy, SKA-DPO, guided by a skeleton-mesh alignment score, to further improve geometric fidelity. Together, our framework enables a more intuitive workflow for creating ready to animate 3D content. To the best of our knowledge, our work is the first to generate rigged 3D meshes conditioned on user-drawn 2D strokes. Extensive experiments demonstrate that Stroke3D produces plausible skeletons and high-quality meshes.