Order Matters: 3D Shape Generation from Sequential VR Sketches

AI summary

Overview

• Research area: Computer vision and 3D generative modeling, specifically sketch-conditioned 3D shape synthesis from virtual reality (VR) sketches.
• Technical level: Advanced. The paper assumes familiarity with transformers/BERT-style encoders, latent diffusion models, signed distance functions (SDFs), and vector-quantized autoencoders (VQ-VAE).
• Scope: The paper introduces VRSketch2Shape, described as the first framework and multi-category dataset for generating 3D shapes from sequential VR sketches, where stroke and point ordering are explicitly modeled rather than discarded.

What This Paper Is About

VR sketching lets designers draw directly in 3D, but existing sketch-to-shape models treat a VR sketch as an unordered point cloud, throwing away the order in which strokes and points were drawn. The authors argue this temporal ordering encodes connectivity, structure, and design intent, and they build both a dataset and a model that preserve it. The goal is to turn a sequence of hand-drawn 3D strokes into a faithful 3D shape.

Key Contributions

1. An automated, learning-free pipeline that generates synthetic sequential VR sketches from arbitrary 3D shapes, producing over 20k paired samples in roughly 10 hours on a standard workstation (20,838 samples are reported).
2. The VRSketch2Shape dataset, described as the first open-access, multi-category collection of sequential VR sketches, with 20,838 synthetic sketch–shape pairs plus 900 hand-drawn sketches across four categories (300 chairs, 200 tables, 200 cabinets, 200 airplanes), collected from 15 participants at an average of 15 minutes per sketch (approximately 225 person-hours).
3. An order-aware sketch encoder — a modified BERT architecture that tokenizes a sketch as ordered strokes and points — coupled with SDFusion, a diffusion-based 3D generator, trained end-to-end in a single stage.
4. Empirical evidence that ordering matters, including ablations showing large drops when stroke/point order, augmentations, or synthetic pretraining are removed, and showing the sequential formulation beats point-cloud and multi-view image encodings.

Main Findings

• Order is the decisive design choice. In a zero-shot synthetic-to-real setting on the chair subset, the full model reaches F-score 56.8 with CD ×1000 of 5.1, while removing stroke and point indices ("w/o stroke ordering") drops it to F-score 48.9 with CD 7.1.
• Synthetic pretraining is essential. Skipping pretraining and training only on the 200 real chair sketch–shape pairs collapses the model to F-score 12.0 with CD ×1000 of 99.9.
• Augmentations help. Disabling augmentations degrades the model to F-score 49.0 with CD 6.6, versus 56.8 / 5.1 for the full model.
• Sketch format matters. Encoding sketches as point clouds with PointNet++ yields F-score 30.8 and CD ×1000 of 25.8; encoding them as rendered 2D images yields F-score 23.8 and CD 62.6.
• The encoder design matters. Replacing the proposed encoder with a direct 3D extension of SketchBERT gives F-score 50.4 and CD 6.3, below the full model.
• State-of-the-art on 3DVRChair. On the public 3DVRChair benchmark (1,005 real chair sketch–shape pairs; official split of 803 training and 202 evaluation), the method achieves F-score 31.1 and CD ×1000 of 25.8, against Luo et al. at 26.6 / 35.5 and LAS-diffusion (which uses 2D renders of sketches) at 26.1 / 66.0. The authors state this is a CD reduction of more than 60% and an F-score improvement of over 40% relative to competing approaches.
• State-of-the-art on VRSketch2Shape. On the new dataset, the method reaches F-score 64.3 on the chair category and 69.8 across all four categories, versus Luo et al. at 42.2 and 48.8, and LAS-diffusion at 37.0 and 40.2. Notably, Luo et al. also perform better on this dataset than on their own, which the authors attribute to the reduced ambiguity of automatically aligned synthetic sketches.
• Few real sketches are enough. Fine-tuning with as few as 50 real sketches per category reaches near-optimal results, and the zero-shot setting (no fine-tuning) is already strong.
• Free-hand sketching works. In a user study with 40 sketches (20 free-hand, 20 reference-guided), 100 sketch–shape pairs (40 of the authors' reconstructions, 40 from Luo et al., and the 20 ground-truth shapes), 38 participants gave 1,900 ratings on a 1–5 Likert scale. The authors' predictions scored 3.92 ± 0.74 with a reference and 3.60 ± 1.01 free-hand, versus 2.76 ± 0.84 and 2.02 ± 0.89 for Luo et al.; reference shapes scored 4.76 ± 0.50.
• Partial sketches are handled well. Given only the first fraction of a sketch sequence (padded with learned MASK tokens), the model infers coherent geometry; the authors report that their order-aware model outperforms an order-agnostic variant by +6.6 F1-score when reconstructing from the first half of a human-drawn sketch, and reaches near-maximum performance with only the first half of drawn points.
• Ordering consistency, not human-likeness, is what matters. Reversing stroke order has little effect (F-score 56.2, CD 5.0), scrambling strokes causes a moderate

Authors’ abstract

VR sketching lets users explore and iterate on ideas directly in 3D, offering a faster and more intuitive alternative to conventional CAD tools. However, existing sketch-to-shape models ignore the temporal ordering of strokes, discarding crucial cues about structure and design intent. We introduce VRSketch2Shape, the first framework and multi-category dataset for generating 3D shapes from sequential VR sketches. Our contributions are threefold: (i) an automated pipeline that generates sequential VR sketches from arbitrary shapes, (ii) a dataset of over 20k synthetic and 900 hand-drawn sketch-shape pairs across four categories, and (iii) an order-aware sketch encoder coupled with a diffusion-based 3D generator. Our approach yields higher geometric fidelity than prior work, generalizes effectively from synthetic to real sketches with minimal supervision, and performs well even on partial sketches. All data and models will be released open-source at https://chenyizi086.github.io/VRSketch2Shape_website.