BrepGaussian: CAD reconstruction from Multi-View Images with Gaussian Splatting

AI summary

Overview

Research area: Computer vision and computer graphics, specifically image-based 3D reconstruction of CAD models (reverse engineering), combining neural rendering (Gaussian Splatting) with parametric B-Rep fitting.

Technical level: Advanced. The paper assumes familiarity with 3D Gaussian Splatting, 2D Gaussian Splatting, boundary representation (B-Rep), parametric primitive fitting, contrastive/triplet learning, and RANSAC.

Scope: The paper presents BrepGaussian, a two-stage framework that reconstructs complete parametric B-Rep CAD models directly from multi-view RGB images, without point cloud supervision.

What This Paper Is About

Recovering a CAD model means inferring the underlying parametric surfaces, edges and topology of an object, not just its visible shape. Existing methods for this task (reverse engineering) almost all take dense, clean point clouds as input and require heavy manual annotation. This paper asks whether the same result can be obtained from ordinary multi-view photographs, using a Gaussian Splatting renderer whose primitives carry learnable edge and patch features, followed by a constraint-guided primitive fitting stage.

Key Contributions

1. BrepGaussian framework: A pipeline for CAD reconstruction from multi-view images that couples 2D Gaussian Splatting with parametric primitive fitting, producing accurate parametric reconstructions. The authors state it is the first framework to reconstruct complete B-Rep CAD models directly from multi-view images without any point cloud supervision.

2. Two-stage learning process: A training scheme in which the first stage learns geometry and edge semantics, and the second stage freezes the geometric parameters and learns patch instance features via contrastive learning, so that complex patch learning does not damage the reconstructed geometry.

3. Constraint-guided primitive fitting module: A fitting procedure that uses the labels obtained from Gaussian training — RANSAC-based fitting of planes, cylinders and spheres, hierarchical extraction of lines, curves and corners, point cloud constraint refinement, bottom-up assembly, and final topological adjustment — to produce a watertight B-Rep.

4. Edge- and patch-aware Gaussian features plus an ellipse-based densification scheme: Each Gaussian carries a scalar edge value (Stage 1) and a high-dimensional feature vector (Stage 2, dimension 16), and Gaussians are sampled into point clouds by taking the center plus four additional points along the ellipse for ellipses whose major axis is not extremely larger than the minor axis.

Main Findings

• Patch segmentation results (Table 1): On densified point clouds reconstructed from multi-view images, BrepGaussian achieves Precision 0.8903, Recall 0.9181 and F1 0.9040, versus PCER-Net at 0.5357 / 0.7924 / 0.6392 and ParSeNet at 0.6229 / 0.2364 / 0.3427. Against methods using ground-truth point clouds, SED-Net scores 0.9490 / 1.0000 / 0.9738, PCER-Net 0.8760 / 0.9119 / 0.8936, HPNet 1.0000 / 0.2142 / 0.3528, and ParSeNet 0.5108 / 0.2646 / 0.3486. The paper states the authors' method outperforms PCER-Net but scores lower than SED-Net when compared against ground-truth-point-cloud inputs.

• Edge segmentation results (Table 2): BrepGaussian reaches Precision 0.9350, Recall 0.9253, F1 0.9301, against PCER-Net on densified points at 0.7149 / 0.8349 / 0.7703, PCER-Net on ground-truth points at 0.8807 / 0.9563 / 0.9169, and SED-Net on ground-truth points at 0.9693 / 0.9913 / 0.9802.

• CAD reconstruction results (Table 3): With D_c (10^-2) for surface and curve, and D_h (10^-1) for surface and curve, BrepGaussian reports 4.90 / 5.01 and 3.351 / 3.626. Point2CAD with the authors' labels reports 3.38 / 5.42 and 2.413 / 3.858; Point2CAD with PCER-Net labels reports 7.08 / 20.45 and 3.394 / 7.276; Split-and-Fit on densified points reports 6.23 / 13.98 and 3.523 / 4.962. The paper notes the authors' method achieves the best performance on curve reconstruction and slightly lower surface scores than Point2CAD with their labels, but attributes Point2CAD's higher scores to redundant patch reconstruction and inferior qualitative quality.

• Segmentation metric definition: Precision and Recall are computed by geometric matching with threshold τ = 0.08 (unit length), and F1 is the harmonic mean of the two. Surface and edge quality are measured with Chamfer Distance (CD) and Hausdorff Distance (HD).

• Ablation on training scheme (Table 4): Removing two-stage learning drops patch segmentation to 0.8709 / 0.6805 / 0.7672 (Precision / Recall / F1); replacing the triplet loss with a pairwise contrastive loss gives 0.8395 / 0.9374 / 0.8857; the full model gives 0.8903 / 0.9181 / 0.9040.

• Ablation on reconstruction (Table 5): Without edge segmentation, results are 4.63 / 5.83 and 3.601 / 3.705 (D_c surface / curve in 10^-2, D_h surface / curve in 10^-1); without densification, 5.99 for surface D_c and 4.184 for surface D_h; the full model gives 4.90 / 5.01 and 3.351 / 3.626.

• Mask quality (Table 6): Manual correction yields Precision 0.8704, Recall 0.9191, F1 0.8941 and D_c 5.04; fine-tuned SAM yields 0.8896, 0.8363, 0.8621 and 5.33; vanilla SAM yields 0.8435, 0.8090, 0.8259 and 5.66. The authors report the pipeline can be made fully automatic with a minor loss of accuracy.

• Number of input views: Models trained with 20, 30, 40 and 50 views show that 30 views or fewer produce noisy patch labels and incomplete reconstructions, while 50 views are required for clean, dense, high-quality B-Rep reconstructions.

• Real-world evaluation: BrepGaussian is also tested on real-world scenes from the ABO dataset and on smartphone photos, and the paper reports it produces clean CAD models.

• Data preparation cost: Because ABC-NEF images are low-texture, direct SAM use gives inconsistent and fragmented masks across views; the authors refined each object manually with a script, spending around three minutes per object.

Methodology in Plain English

The pipeline starts with multi-view RGB images of a CAD object. An edge detector produces 2D edge masks, and those masks are used as prompts to the Segment Anything Model (SAM) to produce per-patch masks for each view.

The rendering backbone is 2D Gaussian Splatting, in which every Gaussian is a flat, oriented elliptical disk lying on the surface rather than a volumetric blob — a choice the authors argue suits CAD geometry dominated by planes and low-curvature surfaces. Each Gaussian is extended with learnable features on top of the usual position, covariance, color and opacity.

Training happens in two separate stages. In Stage 1, each Gaussian gets a single scalar edge value in [0, 1], rendered with the same alpha-compositing weights used for color, and the network jointly learns the object geometry and its edge semantics using a geometry loss (an L1 plus D-SSIM combination) and an edge-aware loss, weighted as L_geo + 0.1 * L_edge. In Stage 2, the geometric parameters such as positions and spherical harmonics are frozen so that patch learning cannot harm the geometry; each Gaussian instead gets a higher-dimensional feature vector, and a triplet loss with cosine distance pulls features from the same 2D patch mask together while pushing features from different masks apart, selecting the hardest negative sample from other masks. This handles the fact that masks across views are not consistent with each other.

The trained Gaussians are then turned into a labeled point cloud: each Gaussian contributes its center point, and ellipses whose major axis is not extremely larger than their minor axis contribute four more points sampled along the ellipse. This mirrors the observation that edge regions need many elongated Gaussians while flat regions need only a few nearly spherical ones.

Finally, a constraint-guided fitting module assigns each patch a primitive type — plane, cylinder or sphere — and fits it with RANSAC, which remains stable even on noisy points. Intersections between fitted primitive pairs give candidate lines and curves, and the edge point cloud restricts these to valid segments by projecting the edge points onto the line or curve to determine the parameter range t. Corner candidates come from intersections of three planes or two lines and are clustered into final corners. Surfaces are then refined under the constraints of the line and curve segments, and Boolean operations assemble everything into a clean, watertight B-Rep.

Why This Matters

The work targets a practical bottleneck in reverse engineering: high-quality point clouds are expensive to acquire and require tedious annotation, whereas images are cheap and scalable. If CAD models can be recovered directly from photographs, the gap between widely available image data and parametric 3D modeling narrows considerably. The paper reports that its method outperforms baselines given point clouds reconstructed from the same multi-view images, and claims to be the first to produce complete B-Rep reconstructions from images with no point cloud supervision.

Real-world applications:

• Reverse engineering of legacy parts: Turning photographs of existing mechanical parts into editable parametric CAD models with planes, cylinders, spheres, curves and corners.
• Digital twins and as-built documentation: Rebuilding CAD representations of structures and industrial equipment from camera imagery; one co-author is affiliated with a tunnel and bridge intelligent management company, pointing at civil infrastructure.
• Consumer and e-commerce 3D content: Generating structured CAD assets from smartphone photos, which the paper demonstrates in its real-world scene experiment.
• Robotics and simulation: Providing topology-consistent, watertight geometry for downstream simulation and manipulation tasks where meshes are insufficient.

Industry relevance: CAD vendors, manufacturing and metrology software, AEC/infrastructure inspection, and any pipeline that currently requires scanning hardware plus manual segmentation could benefit from an image-only front end. Adoption is constrained by the current pipeline's dependence on mask quality and its reported complexity.

Future Directions

• Removing the manual mask correction step. The authors spent roughly three minutes per object refining masks, and their fine-tuned SAM variant trades F1 (0.8621) and D_c (5.33) for automation, compared with manual correction at F1 0.8941 and D_c 5.04. Closing that gap fully is an open problem.
• Reducing the view requirement. The experiments indicate that 30 views or fewer yield noisy patch labels and incomplete reconstructions, so lowering the number of input views needed for clean B-Rep output is a clear next target.
• Extending to complex objects. The stated limitation is that point and label quality from multi-views limits capability on complex objects; improving robustness there is an explicit avenue.
• Simplifying the pipeline. The authors describe the overall pipeline as still relatively complex, suggesting integration or streamlining of the edge detection, SAM prompting, two-stage Gaussian training and fitting stages.
• Generalizing beyond planes, cylinders and spheres. The primitive fitting module covers these three parametric surface types, leaving open how other surface families would be handled.

Target Audience

Researchers and graduate students in computer vision, computer graphics and geometric modeling who work on 3D reconstruction, neural rendering or CAD reverse engineering. It will also interest industrial R&D engineers in CAD, manufacturing and infrastructure inspection evaluating image-based alternatives to point-cloud scanning pipelines. Readers need prior familiarity with Gaussian Splatting and parametric surface fitting to follow the method and metrics comfortably.

Authors’ abstract

The boundary representation (B-rep) models a 3D solid as its explicit boundaries: trimmed corners, edges, and faces. Recovering B-rep representation from unstructured data is a challenging and valuable task of computer vision and graphics. Recent advances in deep learning have greatly improved the recovery of 3D shape geometry, but still depend on dense and clean point clouds and struggle to generalize to novel shapes. We propose B-rep Gaussian Splatting (BrepGaussian), a novel framework that learns 3D parametric representations from 2D images. We employ a Gaussian Splatting renderer with learnable features, followed by a specific fitting strategy. To disentangle geometry reconstruction and feature learning, we introduce a two-stage learning framework that first captures geometry and edges and then refines patch features to achieve clean geometry and coherent instance representations. Extensive experiments demonstrate the superior performance of our approach to state-of-the-art methods. We will release our code and datasets upon acceptance.