Parametric CAD Bench v3: Evaluating Agentic Parametric CAD Authoring from Text and Engineering Drawings

AI summary

Overview

Research area: AI agent evaluation, specifically agentic computer-aided design (CAD) authoring — generating native parametric CAD models from natural-language descriptions and from engineering drawings.

Technical level: Advanced. The paper assumes familiarity with parametric CAD concepts (feature trees, sketches, pads/pockets, boundary representations) and with agent-evaluation methodology (sandboxed runtimes, programmatic graders, reward aggregation).

Scope: A 100-task benchmark and dated 13-system leaderboard measuring whether general-purpose AI agents can produce geometrically correct, specification-consistent, native FreeCAD documents across three CAD workflows.

What This Paper Is About

Producing a CAD script that merely executes does not prove an agent satisfied an engineering design request — the geometry, the specified dimensions, and the native modeling history all matter. The authors build Parametric CAD Bench v3, a 100-task suite in a pinned FreeCAD environment where agents must submit native CAD documents to a separate programmatic verifier, rather than a CadQuery script or an exported shape. The goal is to measure complete model–agent systems on three workflows — text-to-CAD creation, create-and-edit, and reconstruction from engineering drawings — using a continuous reward that combines geometric similarity with dimension and specification consistency.

Key Contributions

1. A versioned 100-task evaluation suite spanning three CAD workflows with native parametric output requirements: 30 text-to-CAD creation tasks, 30 text-to-CAD create-and-edit tasks, and 40 engineering-drawing reconstruction tasks.
2. An isolated execution and programmatic scoring protocol, with frozen configurations (FreeCAD 1.1.0, Harbor 0.23.0, task schema 1.3, verifier gnucleus-freecad-validator 0.6.0 with its v2 scorer), held-back reference assets placed in a separate verifier image, and per-task diagnostic artifacts.
3. A dated evaluation of complete model–agent systems (a September 24, 2026 snapshot covering 13 systems), reporting differences across workflows and observed failure modes rather than a single binary success rate.
4. A reward formulation that replaces binary pass/fail with a harmonic combination of geometry similarity and CAD-specification consistency, so that a zero on a required axis or deliverable cannot be rescued by another axis.

Main Findings

• Top overall system: Claude Code with Claude Opus 5.5 has the highest overall point estimate at 61.03%, followed by mini-swe-agent with the same model at 57.96%, Codex with GPT-6 Astra at 56.87%, and Claude Code with Claude Fable 5.1 at 56.75%. The authors state that point-estimate ordering alone does not demonstrate a statistically decisive ranking.
• Workflow leaders differ: Fable 5.1 with Claude Code leads creation (55.48%), Astra with Codex leads create-and-edit (44.37%), and Opus 5.5 with Claude Code leads image-to-CAD (84.66%).
• Model upgrade helps unevenly: Within Claude Code, Opus 5.5 raises the overall estimate from Opus 5's 50.86% to 61.03%, a 10.18-point increase with a reported paired 95% interval of 5.22–15.13 points. Per-task counts: creation 8 higher / 18 similar / 4 lower; create-and-edit 4 / 23 / 3; image-to-CAD 22 / 15 / 3.
• Drawing reconstruction drives the gain: The image-to-CAD mean improves by 22.02 points (median paired drawing-task gain 7.62 points), versus 1.39 points for creation and 3.17 for create-and-edit. All five drawings that scored zero under Opus 5 score positive under Opus 5.5, and the 22.02-point image gain accounts for approximately 87% of the overall improvement after the 30/30/40 workflow weights.
• Agent framework effect is not established overall: With Opus 5.5 held constant, Claude Code exceeds mini-swe-agent by 3.08 points overall, but the paired interval is [−1.15, 7.31]. Claude Code leads creation by 8.07 points (9 higher / 20 similar / 1 lower), mini-swe-agent leads create-and-edit by 1.68 points (2 / 24 / 4), and the image-to-CAD gap of 2.90 points comes from outcome magnitudes (5 higher / 30 similar / 5 higher). Both Opus 5.5 systems exceed 80% on drawing reconstruction (84.66% and 81.77%).
• Multimodal CAD separates systems most sharply: Across the 13 rows, image-to-CAD means span 8.18% to 84.66% (76.49 points), versus 23.32% to 55.48% (32.16 points) for creation and 23.23% to 44.37% (21.14 points) for create-and-edit. The five highest image-to-CAD systems reach 62.64–84.66%; the remaining eight reach only 8.18–31.10%. Gemini 3.8 Flash scores 31.10% on image-to-CAD with mini-swe-agent's multimodal bridge versus 17.45% with Antigravity, a 13.65-point gap.
• Best observed reward-cost balance: Codex with GPT-6 Astra has the lowest cost-per-score-point ratio, $317.81 / 56.87 points = $5.59 per point, and the shortest mean elapsed trial duration at 14.42 minutes (versus 16.58 for GPT-5.6 Sol with Codex, 31.11 for Claude Code with Fable 5.1, and 32.51 for Claude Code with Opus 5.5). Astra's 56.87% is close to Fable 5.1's 56.75% at roughly 70% lower known API cost and less than half the mean trial duration; the higher-scoring Opus 5.5 system gains 4.17 points over Astra at 1.81× the API cost and 2.25× the mean trial duration.
• Failures concentrate in two-stage construction: Of 1,300 selected final trials, 393 (30.23%) earn zero reward. Create-and-edit has the highest zero rate at 41.54%, versus 22.56% for creation and 27.50% for image-to-CAD. 24 tasks are zero for at least 7 systems (5 creation, 11 create-and-edit, 8 image), and 5 tasks are zero for all 13 systems, all in create-and-edit. 26 tasks have no system reaching 0.5, while 15 tasks reach a perfect reward of exactly 1.0 in at least one system.
• Zeros are mostly scoring failures, not crashes: Of the 393 zeros, 326 involve a zero geometry component (including 50 that also have zero specification), 33 fail specification consistency only, and the remainder are 23 final exceptions, 7 missing outputs, and 4 scorer crashes.
• Two example tasks fail on opposite axes: On the trussed plate (freecad-2002df3ce7, 86 key parameters), 9 of 13 systems score zero, eight of them on specification alone despite geometry similarity between 0.126 and 0.445; the best result is Fable 5.1 with Claude Code at 0.999 with every parameter consistent. On the clevis connector (freecad-fa0a1abef3, 44 key parameters), 12 of 13 systems score positive, and Claude Opus 5 with Claude Code reaches 1.000; most positive outcomes pass nearly all dimension checks but have geometry similarity only between 0.175 and 0.500.

Methodology in Plain English

Each task runs inside a fixed, sandboxed FreeCAD environment. The agent receives a task prompt — a part description with named parameters, a base description plus an edit instruction, or an engineering drawing — and must save native FreeCAD documents (for create-and-edit, both answer_base.FCStd and answer_edit.FCStd; for drawing tasks, also /app/answer.py and /app/answer.FCStd). Reference geometry and grader assets live in a separate verifier image the agent cannot see.

A programmatic verifier then scores two independent axes. Geometry similarity compares surface-type areas, volume, total surface area, normalized principal moments of inertia, and face-center ICP alignment, with weighted subscores (0.05 surface-type, 0.175 volume, 0.175 area, 0.10 moments, normalized by 0.50) multiplied by (0.50 + 0.50 × q_ICP). Volume earns full credit at relative error of at most 0.1% and zero at 1%; the other error-based subscores use 1% and 10% thresholds with logarithmic interpolation. Gates zero the score for an oriented-bounding-box extent error of 10% or more, for documents without one non-empty PartDesign::Body producing one solid from allowed native features, and for more than 5,000 faces or face/vertex counts differing from the reference by more than 50%. Specification consistency parses the task's held-back parameter list and checks it against measurements from the candidate document using engineer-written param_check.py logic, with a default 1% scalar relative tolerance and a failure budget of 10 parameters. The two axes combine through a harmonic mean that returns zero if either axis is zero; for create-and-edit the harmonic mean is applied twice, once within each stage and once across the base and target artifacts. The reported overall percentage is the mean of 100 per-task rewards, equivalently 0.30 × creation + 0.30 × create-and-edit + 0.40 × image-to-CAD. The authors emphasize that this continuous reward is not Pass@1, and they separately report the fraction of rewards exactly equal to 1.0 as the perfect-task rate.

Why This Matters

Impact on research. The paper argues that "the code ran" and "the render looks similar" are insufficient tests of a CAD agent, because a model can resemble the requested part while carrying wrong dimensions, missing features, or a static shape with no editable history. Requiring native FreeCAD documents with an application-managed feature tree, and scoring geometry and specification separately, gives the field a graded signal rather than a binary one. The dated snapshot and separation of complete model–agent systems from raw model capability also counters the common practice of attributing system-level scores to models alone.

Real-world applications:

• Automating first-pass part modeling from text specifications in mechanical design workflows.
• Reconstructing editable CAD models from legacy or supplier engineering drawings that exist only as dimensioned images.
• Parametric variant generation, where an existing base model is edited to new target dimensions rather than rebuilt.
• Benchmarking and procurement decisions when organizations compare agent stacks on cost, speed, and reliability rather than leaderboard rank alone.

Industry relevance. The benchmark targets the tools industry actually uses — FreeCAD with OCCT, whose commercial counterparts include Siemens NX, CATIA, Creo, and SolidWorks — and explicitly distinguishes its native-document contract from CadQuery-based program synthesis used by BenchCAD. The cost-per-point and per-task duration results give practitioners a concrete basis for choosing between systems, and the finding that create-and-edit remains the weakest workflow identifies where engineering effort is most needed.

Future Directions

• Close the two-stage construction gap: create-and-edit is the weakest workflow (41.54% zero rate, 44.37% best mean) and accounts for all five tasks that no system solves; the paper does not establish whether the bottleneck is feature-history reuse, constraint preservation, or edit-direction reasoning.
• Separate model capability from agent interface: the authors call their comparisons conditional and not fully controlled ablations, and note the mini-swe-agent multimodal bridge (prompt plus a base64-to-image bridge) is part of the measured system; cleaner ablations would isolate image-transport, tool use, and failure recovery as causes of the 76.49-point image-to-CAD spread.
• Extend beyond the current contract: the limitations name assembly mating, FEM/FEA, manufacturing qualification, and broad organic-surface modeling as uncovered, and note that the checks are proxies that do not certify manufacturability or editability under arbitrary future changes.
• Address validity and reproducibility: the paper flags that alternative valid constructions may be penalized by reference-based comparison (congruent geometry built through a different feature history need not receive g = 1), that single official attempts do not establish run-to-run robustness, and that undisclosed provider training data limits contamination claims.

Target Audience

Researchers and engineers working on agentic AI for engineering design, benchmark designers who need a worked example of multi-axis programmatic grading with isolation and versioning, and CAD/PLM practitioners evaluating whether current agent systems can be trusted with native parametric modeling tasks. Readers without CAD background can follow the results but will need familiarity with feature trees and sketches to interpret the failure analysis.

Authors’ abstract

Producing an executable CAD script is not sufficient to establish that an agent has satisfied an engineering design request. Evaluation must also consider the resulting geometry, specified dimensions, and native CAD modeling history. We present Parametric CAD Bench v3, a 100-task benchmark comprising 30 text-to-CAD creation tasks, 30 text-to-CAD create-and-edit tasks, and 40 tasks that reconstruct mechanical parts from engineering drawings. Agents operate in a pinned FreeCAD environment and submit native CAD artifacts to a separate programmatic verifier. The primary metric is mean continuous task reward, combining geometry similarity and specification consistency rather than binary task success. A September 24, 2026 snapshot covers 13 model-agent systems; Claude Code with Claude Opus 5.5 achieves the highest overall point estimate, 61.03%, and an image-to-CAD mean reward of 84.66%. These measurements describe complete systems, not isolated model capabilities. We release versioned tasks, evaluation configurations, and inspectable result artifacts.