Rethinking Human Intent-to-CAD: Parametric CAD Model Generation via Cooperative Multi-Task Alignment and Spatial-Aware Reinforcement Learning

AI summary

Overview

Research area: Multimodal generative modeling for parametric Computer-Aided Design (CAD) — combining vision-language models, code generation, and reinforcement learning. Technical level: Advanced (assumes familiarity with VLMs, supervised fine-tuning, and policy-gradient/GRPO-family RL). Scope: The paper proposes a new dataset (HiCAD) and a two-stage framework for generating executable CadQuery code directly from hand-drawn sketches and/or textual descriptions, without assuming a target CAD model already exists.

What This Paper Is About

Existing image-to-CAD and point-cloud-to-CAD systems follow a reconstruction paradigm: they need renderings, photographs, or 3D scans of a part that already exists. But in early conceptual design the part does not exist yet — designers have only rough sketches and verbal descriptions. The paper redefines the problem as intent-driven generation, where a model must turn sparse, unstructured human input into rigorous, executable parametric CAD code, and it addresses the accuracy, code-validity, and geometric-fidelity failures that result from that shift.

Key Contributions

1. A reformulated task suite. Four human intent-to-CAD tasks are defined over different input combinations: sketch only (I→C), complete text only (T_CTD→C), sketch plus partial text ((I, T_PTD)→C), and sketch plus complete text ((I, T_CTD)→C) — spanning unimodal to multimodal and low to high information density.
2. The HiCAD dataset. Described as the first large-scale dataset jointly aligning hand-drawn sketches, textual descriptions, and parametric CadQuery codes. It is built by restructuring and expanding DeepCAD, using the CadQuery codes created for DeepCAD by the CAD-Recode project, with splits of 82,075 training / 4,536 validation / 4,015 test samples. Duplicates are removed using DINOv3 feature embeddings with a similarity threshold of 0.99, and 100 test-set models are rendered for volunteer-drawn real sketches.
3. A two-stage framework (HiCAD). Stage I, Cooperative Multi-Task Alignment (CMTA), fine-tunes a pretrained VLM on interleaved data from all four tasks. Stage II, Spatial-Aware Reinforcement Learning (SARL), uses Group Sequence Policy Optimization (GSPO) with a reward combining topological consistency (graph edit distance over B-Rep face adjacency graphs), geometric consistency (3D IoU), and an invalid-code penalty.
4. Empirical validation across all tasks. The method reports the best IoU and lowest invalid rate against both general-purpose VLMs and CAD-specific models, including on real human-drawn sketches.

Main Findings

• Best results on all four tasks (Table 1). HiCAD reaches 69.49% IoU on sketch-to-CAD, 74.27% on complete-text-to-CAD, 76.03% on sketch + partial text, and 80.90% on sketch + complete text, with Mean CD of 4.40, 6.91, 3.91 and 3.04 respectively (CD scaled by 10³).
• Code validity improves by an order of magnitude. Invalid rate falls to 0.37% on sketch-to-CAD versus 9.41% for Cadrille and 3.96% for CAD-Coder; the strongest baseline's 9.41% is explicitly cited as reduced to 0.37%. Across tasks, HiCAD's invalid rate stays between 0.17% and 0.42%.
• CAD-specific baselines trail on IoU. Cadrille scores 66.54 / 67.19 / 66.86 / 72.56 IoU and CAD-Coder 48.89 / 60.08 / 56.38 / 64.61 across the four tasks, well below HiCAD.
• General-purpose VLMs fail at geometric precision. GPT-4o scores 20.49% IoU on sketch-to-CAD with a Mean CD of 53.52, and GPT-5-mini only 9.51% IoU; even the best general model on that task (Gemini-2.5 at 23.72%) is far below the trained CAD models. These models were evaluated in a 2-shot setting.
• Performance rises with richer input. HiCAD's IoU climbs monotonically from sketch-only (69.49%) to sketch + complete text (80.90%), while general VLMs degrade and become less valid in multimodal settings.
• Multi-task alignment beats per-task fine-tuning (Table 2). Replacing single-task supervised fine-tuning (63.84% IoU on sketch-to-CAD) with CMTA raises it to 66.87%, and on sketch + partial text from 66.61% to 72.39%.
• SARL adds robustness on top of CMTA. Adding SARL pushes invalid rates down to roughly 0.2%–0.4% and IoU up further (e.g., 69.09% on sketch-to-CAD in the ablation table, which lists 69.09% for the full configuration versus 69.49% in Table 1).
• Joint multi-task RL matters. Experiments varying which tasks are used during SARL show single-task SARL helps each scenario somewhat, but joint SARL across all tasks substantially outperforms it.
• Real human sketches (Table 3). HiCAD achieves 43.27% IoU with 3.00% invalid rate on sketch-to-CAD, versus 35.72% / 8.00% for CAD-Coder and 17.74% / 87.00% for Cadrille; with complete text added it reaches 73.37% IoU with a 0.00% invalid rate. GPT-4o, Gemini-2.5 and Claude-3.7 all fall below 10% IoU on the sketch-only real-input setting.

Methodology in Plain English

Building the data. The authors start from DeepCAD models that already have CadQuery code. For each model they create two synthetic inputs: a hand-drawn-style 2D sketch (rendered without hidden lines, so only visible contours appear, mimicking how people sketch) and a text description. The description is produced by feeding a vision-language model (Qwen3-VL-32B) the CadQuery code, a rendered image, and basic geometric attributes such as length, width, height, and number of through-holes through a prompt template; descriptions cover only geometry, structure, and dimensions, not color or material. To simulate incomplete user input, they also produce "partial" descriptions by randomly deleting a uniformly sampled number of sentences (between 1 and M−1) from the complete description.

Stage I — Cooperative Multi-Task Alignment. A Qwen3-VL-4B-Instruct model is fine-tuned with standard next-token cross-entropy on CadQuery code, mixing batches from all four input configurations rather than training separate models. This forces sketch features and text features into a shared modeling space, so the model can fill in missing information when input is partial. Settings: per-device batch size 8, learning rate 1.0×10⁻⁵ with cosine scheduling, 2 epochs.

Stage II — Spatial-Aware Reinforcement Learning. The fine-tuned model becomes a policy that samples 10 candidate CadQuery programs per query. Each candidate is executed; if it fails, it receives a fixed penalty of −0.1. If it runs, its reward blends two signals: a topological score of 1/(1+GED) computed from face adjacency graphs of the predicted and ground-truth boundary representations using OpenCascade, and the 3D volumetric IoU computed by decomposing multi-body models into non-intersecting submeshes and summing pairwise intersections. Weights are λ₁ = 0.2 for topology and λ₂ = 0.8 for IoU. Updates use GSPO, which computes the importance ratio at the sequence level (length-normalized) and clips the whole generated program rather than individual tokens — a choice aimed at stable training on long code sequences. SARL runs at a reduced learning rate of 1.0×10⁻⁶, built on the EasyR1 framework with DeepSpeed and Flash-Attention, on 4 NVIDIA A100 80GB GPUs.

Evaluation. Geometric accuracy is measured with IoU and Chamfer Distance (mean and median), plus Invalid Rate, using the Cadrille open-source evaluation protocol for all methods.

Why This Matters

The work reframes CAD generation from "reconstruct something that exists" to "create something that does not yet exist," which is the actual situation in conceptual design. It also shows that reward design grounded in executable geometry — rather than token-level likelihood — is what closes the gap between syntactically valid code and geometrically correct parts; the drop in invalid rate from 9.41% to 0.37% is the clearest evidence. Releasing a tri-modal dataset (sketch, text, CadQuery code) gives the community a shared benchmark where previously only image- or point-cloud-conditioned pipelines could be compared.

Real-world applications:

• Rapid concept prototyping: a designer sketches a bracket on a whiteboard and gets an editable parametric model, not just a mesh.
• Text-driven variant generation: describing a part family ("enclosure with four through-holes, 120 mm wide") to produce dimensioned code that can be edited further.
• Sketch-plus-spec workflows where a drawing carries topology while a short text note carries critical dimensions.
• Design assistance for non-experts and CAD training, where the barrier is specialized modeling knowledge rather than the idea itself.

Industry relevance: Output is executable CadQuery/Python that yields editable B-Rep and parameters, which fits existing CAD software paradigms far better than mesh-only generation; CAD vendors and engineering-tooling companies integrating generative assistants would benefit directly. The explicit invalid-rate metric matters commercially, since a model that emits non-compiling code is unusable in a production loop.

Future Directions

• Richer input modalities. The framework handles sketches and text only; extending to photographs, 3D scans, or annotation markup on existing geometry is unexplored.
• Better reward design. The authors state that reward design "still requires further in-depth research" — alternatives to the 1/(1+GED) topological term and the IoU term, and how to weight them (currently 0.2 / 0.8), remain open.
• Dataset scope. HiCAD is derived from DeepCAD, so scaling to more complex industrial assemblies, and evaluating whether multi-task gains hold there, is an open question.
• Robustness to real sketching styles. The real-sketch evaluation covers only 100 volunteer-drawn models across two task settings, which is a narrow probe of generalization; broader human-input studies would strengthen the claims.

Target Audience

Researchers and graduate students working on multimodal generation, code-generating VLMs, and CAD/geometry processing; engineers building generative design assistants or CAD-automation tooling; and practitioners interested in reinforcement learning with execution-based, geometry-aware rewards for structured program synthesis. Readers need comfort with VLM fine-tuning and policy-optimization methods to get the most from the methodology sections.

Authors’ abstract

Parametric Computer-Aided-Design (CAD) modeling from human intent remains challenging, particularly during the conceptual design stage, where design goals are expressed through incomplete and unstructured modalities (e.g., hand-drawn sketches and textual descriptions). In this work, we rethink the human intent-to-CAD pipeline and propose a unified method that directly maps multi-level human intents to executable codes, without assuming the prior existence of target CAD models. To support our study, we construct HiCAD, the first large-scale dataset aligning hand-drawn sketches, textual descriptions, and parametric CAD codes. Based on this, we introduce HiCAD, a two-stage framework comprising Cooperative Multi-Task Alignment to bridge the representational gap between heterogeneous inputs, and Spatial-Aware Reinforcement Learning to enforce geometric and topological consistency. Extensive experiments demonstrate that our method significantly outperforms existing baselines across multiple tasks, validating its effectiveness and robustness in transforming heterogeneous human intents into high-fidelity parametric CAD models. Our project page: https://zqwlearning.github.io/HiCAD.