PartNeXt: A Next-Generation Dataset for Fine-Grained and Hierarchical 3D Part Understanding

AI summary

Overview

Research area: 3D computer vision — large-scale dataset construction and benchmarking for 3D part understanding (part segmentation, part-centric question answering, promptable segmentation).

Technical level: Intermediate. The paper is a dataset and benchmark paper; it assumes familiarity with 3D representations (textured meshes, point clouds), part segmentation metrics such as mIoU and IoU@k, and 3D vision-language models, but the core ideas are described in accessible terms.

Scope in one sentence: The paper introduces PartNeXt, a 23,519-model textured 3D dataset with fine-grained hierarchical part annotations across 50 categories, plus two benchmarks (class-agnostic part segmentation and part-centric 3D question answering) that show current state-of-the-art models struggle with fine-grained part reasoning.

What This Paper Is About

Part-level 3D understanding has largely been driven by PartNet, but PartNet annotations require remeshing (losing textures and deforming geometry) and an annotation interface that demands 3D modeling expertise, which limits scale and usability. The authors build PartNeXt, a dataset of 23,519 high-quality textured 3D models across 50 categories with 350,187 annotated parts organized in hierarchies, annotated directly on textured meshes through a web-based crowdsourcing interface. They then use it to benchmark existing part segmentation methods and 3D LLMs, and to retrain Point-SAM.

Key Contributions

1. A next-generation part dataset. PartNeXt contains 23,519 annotated objects across 50 categories, with 350,187 total annotated parts. Models are sourced from Objaverse (14,811 instances), ABO (2,633), and 3D-FUTURE (6,075). Annotations are made directly on textured meshes, preserving native texture and original geometry — unlike PartNet, which lacks texture and raw geometry due to remeshing.

2. A scalable web-based annotation system. The interface uses a hierarchical annotation workflow, a dual-panel layout (unsegmented mesh on the left, segmented result on the right) that helps with occluded interior parts, and three face-selection tools: connected component selection, bounding-box selection, and per-face selection. AI assistance includes CLIP-based filtering of assets and GPT-4o-assisted hierarchy generation and reference-image creation.

3. Two new benchmarks. (a) Class-agnostic 3D part instance segmentation with 250 evaluation objects (5 per category), using all leaf nodes as ground truth and mIoU as the metric. (b) Part-centric 3D question answering covering part counting, part classification, and part grounding for 3D LLMs.

4. Evidence that annotation source matters. Training Point-SAM on PartNeXt improves performance over training on PartNet, both on PartNet-Mobility and on held-out PartNeXt categories.

Main Findings

• State-of-the-art segmentation methods struggle on fine-grained parts. On the 10 categories reported in the main table, mean IoU is 36.78 for SAMPart3D, 51.57 for SAMesh, and 50.22 for PartField. The paper reports distinct failure modes: SAMesh is relatively strong at fine-grained segmentation but over-segments; PartField occasionally fails to separate adjacent connected regions; SAMPart3D struggles with continuity in weakly-textured areas and shows inconsistent granularity control. The authors state that current methods perform noticeably worse particularly on leaf-level parts.

• Performance varies sharply by category. In the fuller appendix table, PartField reaches 75.37 on Mug and 73.83 on Teapot but only 12.34 on Laptop and 19.51 on Piano; SAMesh reaches 89.98 on Com_Keyboard and 82.59 on Bed but 24.08 on Laptop; SAMPart3D reaches 68.72 on Screwdriver but 6.75 on Guitar and 6.92 on Laptop.

• 3D LLMs show limited part-level ability. Across part counting (mean absolute error) and part classification (accuracy), the reported scores are: with category provided — 3DLLM 2.16 MAE and 0.33 accuracy, PointLLM 1.87 and 0.22, ShapeLLM 1.72 and 0.25; without category — 3DLLM 2.46 and 0.18, PointLLM 1.79 and 0.08, ShapeLLM 1.85 and 0.30. For part grounding, the paper notes that blank entries in the table mean the model failed to respond with a reasonable bounding box; the authors state that performance on part counting and grounding remains limited.

• Training on PartNeXt beats training on PartNet. Evaluating on PartNet-Mobility, Point-SAM trained on PartNet scores IoU@1 39.0, IoU@3 53.7, IoU@5 58.6, IoU@7 60.9, IoU@10 62.9; trained on PartNeXt it scores 40.2, 57.5, 63.2, 65.0, 67.4; trained on a mixture it scores 40.4, 58.3, 64.1, 66.9, 68.7. On the held-out PartNeXt categories, PartNet-trained scores 39.9, 53.9, 58.4, 60.4, 60.3; PartNeXt-trained 44.3, 60.1, 63.2, 64.8, 65.9; mixture-trained 45.3, 61.7, 65.3, 66.6, 67.6.

• Richer category coverage than PartNet and PartObjaverseTiny. The comparison table lists PartNet at 26,671 shapes and 24 categories, PartObjaverseTiny at 200 shapes and 8 categories with no hierarchy, and PartNeXt at 23,519 shapes and 50 categories with texture, raw geometry, semantics, and hierarchy.

• Most annotations fit the predefined hierarchy. The "Other" part distribution is 82.22 percent with zero Other instances, 13.82 percent with one, 1.77 percent with two, 0.30 percent with three, 0.38 percent with four, 0.20 percent with five, and 1.30 percent with more than five.

Methodology in Plain English

The authors assembled models from three public sources and cleaned the Objaverse subset by removing animated models, models with more than 130k faces, and models tagged as scans or architectural objects. Because Objaverse lacks category labels, they defined roughly 100 common categories, encoded category names and Cap3D captions with CLIP's text encoder, assigned each object to the most similar category, discarded anything scoring below 0.75 similarity, and kept the 50 categories with the most objects.

For each category they predefined a part hierarchy guided by five stated criteria: functionality-aware top-level parts, hierarchy following manufacturing sub-parts, enumeration of all part variants under the same parent, atomic leaf nodes, and consistency of function-and-structure definitions across categories. GPT-4o drafted hierarchies and rendered category images were supplied to help cover variants; human experts then reviewed the output. GPT-4o also generated labeled reference images for each part node, supplemented or replaced with curated online images when needed.

Annotation is fully web-based. Annotators expand a collapsible tree, select mesh faces on the untextured-view side, and move them to the segmented side, where each part gets a unique color synced to its hierarchy node. For parts outside the taxonomy, annotators can create an "Other" node at any level. Thirty-five professional annotators did the labeling after a two-day training session, with 5 top-performing annotators handling verification and quality control. Each model took roughly 5 to 6 minutes on average. Every annotation received at least one review, with 5,211 total corrections and a maximum of 8 corrections on a single annotation. The annotation backend ran on a central server with dual Intel Xeon 6326 CPUs; annotators used consumer PCs with Intel Core i5-12400F processors.

Because Cap3D captions proved unreliable for some categories (notably tool-like objects) and some models contained multiple objects, the team temporarily merged related categories into groups such as Tool, Liquid Container, Light, and Computer Device, then decomposed them back into individual categories after annotation.

Why This Matters

Impact on research. PartNeXt supplies textured, hierarchically annotated 3D data at a scale comparable to PartNet (23,519 versus 26,671 shapes) but with twice the category count (50 versus 24) and without the remeshing that strips PartNet of texture and original geometry. The benchmark results give the field a concrete measure of how far current segmentation and 3D LLM methods are from fine-grained part understanding, and the Point-SAM experiments show that the annotation source itself measurably changes downstream model quality.

Real-world applications.

• Robotics manipulation, where a robot needs to distinguish a handle from a lid or a knob from a panel.
• 3D content creation and generative modeling, where parts can be assembled, edited, or replaced individually.
• Affordance and function analysis, such as identifying which object regions can be grasped, opened, or sat on.
• Interactive 3D editing and asset retrieval, where users select or search for specific components within textured models.

Industry relevance. The dataset targets practical bottlenecks: a crowdsourceable annotation interface that avoids expert 3D modeling skills, and part-level labels that retain texture so that visual cues like color and material can be used by both annotators and models. The gains from training Point-SAM on PartNeXt suggest direct value for anyone building interactive segmentation or 3D foundation models, and the part-centric QA benchmark gives a way to evaluate 3D LLMs on structure rather than whole-object recognition.

Future Directions

• Scaling the dataset. The authors state they are actively working to expand beyond the current 23,519 models by incorporating more data from ObjaverseXL.
• Open-vocabulary annotation. Because each category currently needs a predefined fine-grained hierarchy, the dataset cannot easily annotate arbitrary open-vocabulary objects; the authors are exploring deeper integration with vision-language models to relax this constraint.
• Richer part descriptions. PartNeXt currently provides only plain part names per node; adding captions or physical attribute annotations for categories and parts is identified as a way to enrich the data.
• Closing the model gap. The benchmarks show that existing segmentation methods, 3D LLMs, and promptable models all fall short on leaf-level and open-vocabulary part grounding, leaving open the question of what architectures or training regimes can handle fine-grained part structure.

Target Audience

Researchers and engineers working on 3D part segmentation, 3D vision-language models, interactive/promptable 3D segmentation, and 3D dataset construction. It is also relevant to graphics and robotics practitioners who need part-level structure for assembly, generation, or manipulation, and to anyone who has used PartNet and wants textured, hierarchically labeled data without remeshing artifacts. Readers without a background in 3D representations or segmentation metrics will need some grounding in mIoU and IoU@k before the benchmark tables are fully interpretable.

Authors’ abstract

Understanding objects at the level of their constituent parts is fundamental to advancing computer vision, graphics, and robotics. While datasets like PartNet have driven progress in 3D part understanding, their reliance on untextured geometries and expert-dependent annotation limits scalability and usability. We introduce PartNeXt, a next-generation dataset addressing these gaps with over 23,000 high-quality, textured 3D models annotated with fine-grained, hierarchical part labels across 50 categories. We benchmark PartNeXt on two tasks: (1) class-agnostic part segmentation, where state-of-the-art methods (e.g., PartField, SAMPart3D) struggle with fine-grained and leaf-level parts, and (2) 3D part-centric question answering, a new benchmark for 3D-LLMs that reveals significant gaps in open-vocabulary part grounding. Additionally, training Point-SAM on PartNeXt yields substantial gains over PartNet, underscoring the dataset's superior quality and diversity. By combining scalable annotation, texture-aware labels, and multi-task evaluation, PartNeXt opens new avenues for research in structured 3D understanding.