S2AM3D: Scale-controllable Part Segmentation of 3D Point Clouds

AI summary

Overview

Research area: 3D computer vision, specifically part-level segmentation of 3D point clouds using a hybrid 2D–3D learning recipe.

Technical level: Advanced. The paper assumes familiarity with point cloud encoders, tri-plane representations, contrastive learning, transformer cross-attention, and FiLM conditioning.

Scope: The paper proposes a scale-controllable, point-prompted framework (S²AM3D) for segmenting 3D point clouds into parts, backed by a newly curated dataset of more than 100,000 point cloud instances.

What This Paper Is About

Part-level 3D segmentation — breaking a 3D object into its constituent parts rather than just labeling it as a whole — is hard because annotated 3D data is scarce and expensive. Approaches built purely on native 3D models generalize poorly with limited data, while approaches that borrow 2D pre-trained vision knowledge often produce inconsistent results across different viewing angles, especially under occlusions and on thin structures. S²AM3D aims to combine 2D segmentation priors with native 3D supervision so that the resulting per-point features stay globally consistent, and to add a continuous "scale" control that lets a user dial segmentation granularity from fine to coarse.

Key Contributions

1. A 2D–3D training recipe for part segmentation. The method reuses 2D pre-trained knowledge while applying native 3D supervision, producing globally point-consistent part features.

2. A scale-aware prompt decoder. It combines a scale modulator (mapping a continuous scale into FiLM parameters) with bi-directional cross-attention, enabling flexible, real-time adjustment of 3D part segmentation granularity.

3. A scalable data curation pipeline. The pipeline performs part annotation, quality filtering (a trained PointNet validator), and connectivity refinement (DBSCAN-based splitting), collecting over 100,000 labeled point cloud instances for training.

4. Leading benchmark results. S²AM3D reports top performance in both interactive and full segmentation settings on PartObjaverse-Tiny and PartNet-E, and also extends naturally to point-prompted segmentation.

Main Findings

• Interactive segmentation gains. On PartObjaverse-Tiny / PartNet-E, S²AM3D reaches IoU of 46.47 / 62.52 (average 54.50) without a scale prompt, versus 31.46 / 50.23 (average 40.85) for Point-SAM and 35.05 / 39.98 (average 37.52) for P³-SAM.
• Scale prompts help substantially. Adding a scale prompt raises S²AM3D to 61.19 / 77.51 (average 69.35), a performance gain of 14.72% and 14.99% on the two datasets respectively.
• Full segmentation results. S²AM3D achieves 63.29 mIoU on PartObjaverse-Tiny and 77.98 mIoU on PartNet-E (average 70.64), compared with 58.10 / 65.39 (average 61.75) for P³-SAM, 51.54 / 59.10 (average 55.32) for PartField, 48.79 / 56.17 (average 52.48) for SAMPart3D, 20.76 / 21.69 (average 21.23) for Find3D, and 26.66 on PartNet-E for SAMesh.
• 3D supervision is the biggest single contributor. Removing 3D contrastive supervision drops the average from 69.35 to 59.03 (with scale) and from 54.50 to 48.27 (no scale) — the largest degradation among all ablated variants. The paper attributes this to blurry boundaries and internal inconsistencies in the extracted features.
• The curated data matters. Training on PartNet instead of the curated dataset lowers the average to 59.62 (with scale) and 50.34 (no scale), which the authors attribute to richer geometric variation and more diverse part composition in their data.
• Scale embeddings improve robustness even without a scale input. A model trained without scale embeddings scores an average of 50.30 (no scale) versus 54.50 for the full model without scale input.
• Robustness to point density. Results are nearly identical at 10k and 100k input points: 46.47 / 46.46 on PartObjaverse-Tiny and 62.52 / 62.41 on PartNet-E without scale; 61.19 / 60.98 and 77.51 / 77.67 with scale.
• Robustness to scale error. On PartObjaverse-Tiny, perturbations within ±20% change IoU by less than 1.0; at |δ| between 30% and 50% the drop is gradual (−0.66 to −3.56 for underestimates, −1.48 to −3.56 for overestimates); at 100%–300% the drops become more pronounced (down to −20.19 and −18.98).
• Model size and speed. S²AM3D uses 120M parameters and roughly 3 ms per prompt, compared with 311M / ~5 ms for Point-SAM and 112M / ~3 ms for P³-SAM, measured on a single NVIDIA H20 GPU.
• Only XYZ input required. Point-SAM uses coordinates plus color, and P³-SAM uses coordinates plus surface normals, while S²AM3D uses only XYZ coordinates.

Methodology in Plain English

The framework is built in two separately trained stages.

Stage 1 — Building consistent per-point features. A voxel-based encoder (PVCNN) turns the input point cloud into a tri-plane representation (xy, yz, zx planes), which transformer blocks aggregate. During training, these tri-planes are rendered from random viewpoints and supervised by distilling a 2D segmentation model (SAM). Because 2D distillation alone creates cross-view inconsistencies, the authors add a native 3D contrastive loss computed on labeled point clouds. Contrastive pairs are restricted to within a single object (each mini-batch contains one object) so that positive and negative points come from the same instance, avoiding cross-instance semantic mismatches. Points with the same part label attract each other; points with different labels repel. The encoder is initialized from PartField pre-trained parameters.

Stage 2 — Decoding with a prompt and a scale. The encoder is frozen, and a decoder is trained on top. Given a point prompt and an optional continuous scale value between 0 and 1 (the part's point count divided by the total), the scale is converted into a sinusoidal embedding and then into channel-wise FiLM parameters that modulate the global feature map, interleaved with transformer blocks. During training, the scale is randomly dropped out with probability 0.1, which makes FiLM reduce to the identity so the model still works without a scale input. The prompted point feature then interacts with the global features through bi-directional cross-attention (the prompt attends to the cloud, then the cloud attends back to the updated prompt) over four layers, followed by an MLP and a sigmoid to produce per-point probabilities.

Training objective. Because each training sample marks only one part as positive, plain binary cross-entropy biases toward the background. The authors use a hybrid loss: a dynamically reweighted BCE (with a weight adaptively computed from each sample's positive ratio) plus a Dice term that directly optimizes set-level overlap.

Data curation. The dataset is built from Objaverse assets. Surface-area-proportional sampling assigns part labels to points, following the annotation-mining strategy of PartCrafter. To remove bad labels, a binary PointNet validator is trained on 800 manually curated shapes (400 valid, 400 invalid, 70/30 split) and applied to the whole dataset with a confidence threshold of 0.8, achieving precision/recall of 1.00/0.78 on its test set. Connectivity refinement then runs DBSCAN with a radius equal to 0.15 times the axis-aligned bounding-box diagonal of each label, splitting spatially disconnected regions into distinct parts. Objects with 2 to 50 parts are retained.

Why This Matters

Impact on research. The paper argues that the two dominant paradigms — native 3D models and 2D-knowledge transfer — each have structural weaknesses: limited generalization from scarce 3D labels, and cross-view inconsistency from 2D priors. S²AM3D demonstrates that combining 2D priors with native 3D contrastive supervision can produce globally consistent features, and that an explicit continuous scale signal gives granularity control that clustering-based methods (PartField, SAMPart3D) and point-prompt methods (Point-SAM, P³-SAM) do not offer. The curated dataset (over 100,000 instances, around 1.2 million part labels, more than 400 categories, averaging 11 parts per object, with part counts spanning 2–50 in a long-tailed distribution) is positioned as one of the largest publicly available 3D part segmentation datasets and a distributional complement to existing benchmarks.

Real-world applications (several drawn from the paper's own framing):

• 3D content creation, where parts must be generated, replaced, assembled, or parametrically edited.
• Robotic manipulation, which needs part-level rather than object-level understanding.
• Reverse engineering of existing objects into editable part structures.
• Interactive scene editing, where a user clicks a point and dials a granularity to isolate a specific component.

Industry relevance. The method reaches comparable performance to P³-SAM with far less training data, and does so with 120M parameters at roughly 3 ms per prompt — comparable in cost to the smaller P³-SAM baseline and cheaper than Point-SAM's 311M parameters. That efficiency profile matters for interactive tools and for pipelines that need to process large asset libraries.

Future Directions

• Richer prompt modalities. The authors state that future work will explore incorporating text instructions alongside point prompts and scale signals to support more intuitive semantic interaction.
• Failure characterization. The paper's failure analysis section is present but its content is cut off in the available text; the specific failure modes and the "frequent absence and low quality of textures" issue it begins to describe are not reported in full.
• Scale estimation without ground truth. The main experiments set the scale prompt to the ground-truth part size for controlled evaluation, and the paper notes that in practice the scale would come from a user estimate or an external module. How to source reliable scale signals automatically remains an open question.
• Performance under large scale ambiguity. Perturbations of 100%–300% in the scale prompt cause substantial IoU drops (down to −20.19 and −18.98), since the perturbed scale corresponds to a different semantic granularity than the original annotation — leaving an open problem for cases where only a coarse scale hint is available.

Target Audience

Researchers and graduate students working on 3D computer vision, point cloud understanding, or multimodal 2D-to-3D transfer. It is also relevant to practitioners building 3D content creation tools, robotics perception systems, or reverse-engineering applications who need part-level segmentation with interactive granularity control, and to groups looking for a large labeled point cloud part segmentation dataset for their own training.

Authors’ abstract

Part-level point cloud segmentation has recently attracted significant attention in 3D computer vision. Nevertheless, existing research is constrained by two major challenges: native 3D models lack generalization due to data scarcity, while introducing 2D pre-trained knowledge often leads to inconsistent segmentation results across different views. To address these challenges, we propose S2AM3D, which incorporates 2D segmentation priors with 3D consistent supervision. We design a point-consistent part encoder that aggregates multi-view 2D features through native 3D contrastive learning, producing globally consistent point features. A scale-aware prompt decoder is then proposed to enable real-time adjustment of segmentation granularity via continuous scale signals. Simultaneously, we introduce a large-scale, high-quality part-level point cloud dataset with more than 100k samples, providing ample supervision signals for model training. Extensive experiments demonstrate that S2AM3D achieves leading performance across multiple evaluation settings, exhibiting exceptional robustness and controllability when handling complex structures and parts with significant size variations.