LangHOPS: Language Grounded Hierarchical Open-Vocabulary Part Segmentation

AI summary

Overview

Research area: Computer vision — open-vocabulary object–part instance segmentation and zero-shot part-level semantic segmentation, using a Multimodal Large Language Model (MLLM) to ground object–part hierarchies in language.

Technical level: Advanced. The paper assumes familiarity with instance/panoptic segmentation architectures (queries, decoders, pixel decoders, backbones), vision-language models, mAP and mIoU/hIoU metrics, and LoRA-style adaptation of large models.

Scope: The paper proposes LangHOPS, a framework that jointly detects and segments object and part instances from open-vocabulary candidate categories by representing object–part hierarchies in language space and parsing them with an MLLM, evaluated across in-domain and cross-dataset object–part instance segmentation and zero-shot part semantic segmentation.

Paper metadata as reported: arXiv:2510.25263v3 [cs.CV], dated 12 Jan 2026 in the provided content; listed publication date 2025-10-29. Authors are affiliated with INSAIT, Sofia University "St. Kliment Ohridski"; Xi Wang is additionally affiliated with ETH Zurich and TU Munich; Fabien Despinoy with Toyota Motor Europe. Funding is reported from Toyota Motor Europe and the Ministry of Education and Science of Bulgaria.

What This Paper Is About

Prior part-segmentation systems find parts by grouping visual features heuristically or with learned grouping modules, which makes it hard to know which parts belong to which object and hard to generalize to part names never seen during training. LangHOPS instead takes an image plus user-supplied candidate object–part categories, expresses the object–part hierarchy as language, and asks an MLLM to break each segmented object down into its parts. The goal is a single model that handles both object and part instances jointly, across open-vocabulary categories, in-domain and on unseen datasets.

Key Contributions

1. An MLLM-based framework for open-vocabulary object–part instance segmentation. The authors describe LangHOPS as the first framework of its kind to put an MLLM inside the object–part parsing pipeline, so it can use the model's knowledge and reasoning to link concepts across granularities within a hierarchy.

2. Language-grounded object–part hierarchies instead of visual grouping. Rather than heuristic or learnable visual grouping, the object–part hierarchy is formulated in language space, which the authors argue improves both context awareness and parsing accuracy.

3. MLLM-driven part query refinement. Initial part queries derived from object queries are refined by the MLLM before being forwarded to the part decoder, creating a joint training path (gradient flow) between object and part instance segmentation.

4. Extensive evaluation with ablations. Reported results span in-domain and cross-dataset object–part instance segmentation (PartImageNet, Pascal-Part-116 / PPS-116), zero-shot part segmentation (PPS-116, PartImageNet, ADE20K), plus ablations on hierarchy design, MLLM versus Q-Former parsing, one-stage versus two-stage training, prompt ordering/wording robustness, noisy hierarchies, query counts, and backbone finetuning.

Main Findings

• Headline claims in the abstract: LangHOPS surpasses previous methods by 5.5% Average Precision (AP) in-domain and 4.8% cross-dataset on the PartImageNet dataset, and by 2.5% mIOU on unseen object parts in ADE20K (zero-shot). The cross-dataset 4.8% figure matches the gap between LangHOPS (26.7 AP) and PSALM† (21.9 AP) in the Pascal-Part-116 + INS + PART → PartImageNet row of Table 1.

• PartImageNet results (Table 1): Trained on Pascal-Part-116 and evaluated on PartImageNet, LangHOPS reaches obj 44.5, part 8.86, AP 16.7; with +INS it reaches 60.5 / 11.4 / 22.3; with +INS+PART it reaches 62.8 / 16.4 / 26.7 (marked +7.5 part, +10.0 AP relative to the PPS-116 baseline). In-domain on PartImageNet: obj 83.9, part 49.2, AP 56.9. Baselines: PartGLEE 81.4 / 41.5 / 50.4 and PSALM† 79.2 / 40.1 / 48.7 in-domain; with +INS+PART, PartGLEE 61.0 / 9.57 / 21.0 and PSALM† 58.6 / 11.6 / 21.9.

• Pascal-Part-116 results (Table 2): Trained on PartImageNet and evaluated on PPS-116, LangHOPS reaches obj 9.57, part 2.20, AP 3.32; +INS: 23.3 / 3.64 / 6.63; +INS+PART: 22.6 / 4.67 / 7.39 (+4.1 AP over the PPS-116 baseline, versus +3.2 and +3.3 for the baselines as reported). In-domain on PPS-116: obj 54.6, part 15.0, AP 21.0. The text states this direction (PartImageNet → PPS-116) is more challenging for all methods because PPS-116 contains more novel object categories and finer-granularity parts.

• Scalability with part-level data: The gain from adding object-level datasets (INS) is similar across methods, but adding part-level datasets (PART: ADE20K, SA1B, PACO) gives LangHOPS a larger boost. PartGLEE's part-level gain is reported to drop from +5.9 to +5.4 in that setting, while LangHOPS gains +7.5 part mAP and +10.0 AP on PartImageNet.

• Zero-shot part segmentation (Table 3, hIoU): LangHOPS obtains hIoU 52.1 on PPS-116, 72.8 on PartImageNet, and 49.5 on ADE20K. It reports the highest mIoU_seen on all three datasets (59.2, 71.9, 49.3 respectively). It is best on PPS-116 and PartImageNet (PartCATSeg: 50.4 and 72.7) and second-best on ADE20K behind PartCATSeg (50.0). On unseen parts it reports 46.5 (PPS-116), 73.7 (PartImageNet), and 49.7 (ADE20K). The authors note that this benchmark measures open-vocabulary semantic segmentation of parts, which they say is not LangHOPS's core application, and that most competing methods, including PartCATSeg, are designed specifically for the OV-Part benchmark.

• Object–part synergy (Table 4): With part loss removed ("Obj Seg"), part mAP is near 0 (0.00 on PPS-116, 2.08 on PartImageNet). With gradients from the MLLM-based parsing module cut ("Detached Obj-Part Seg"), object mAP drops by 0.6% on PascalPart116 and by 5.0% on PartImageNet relative to "Obj Seg". Joint training ("Obj-Part Seg") improves over the detached setting by 5.4% on object and 1.7% on part segmentation.

• Attention evidence (Table 5): Average attention scores (defined in Eq. 2 as the normalized cosine similarity between queries and dense decoder features, summed over true-positive pixels inside ground-truth masks) rise from 0.76 → 0.82 for objects and 0.58 → 0.67 for parts when moving from detached to synergized object–part segmentation.

• MLLM parsing matters (Table 7): Replacing the MLLM parsing module with a Q-Former ("w/o MLLM") gives 23.2 mAP on PartImageNet and 18.4 on PPS116, versus 26.7 and 19.8 for LangHOPS.

• Language-grounded hierarchy matters (Table 7): Replacing the hierarchy representation with N learnable queries ("w/o hierarchy") gives 22.5 on PartImageNet and 19.1 on PPS116, versus 26.7 and 19.8 for LangHOPS.

Authors’ abstract

We propose LangHOPS, the first Multimodal Large Language Model (MLLM) based framework for open-vocabulary object-part instance segmentation. Given an image, LangHOPS can jointly detect and segment hierarchical object and part instances from open-vocabulary candidate categories. Unlike prior approaches that rely on heuristic or learnable visual grouping, our approach grounds object-part hierarchies in language space. It integrates the MLLM into the object-part parsing pipeline to leverage its rich knowledge and reasoning capabilities, and link multi-granularity concepts within the hierarchies. We evaluate LangHOPS across multiple challenging scenarios, including in-domain and cross-dataset object-part instance segmentation, and zero-shot semantic segmentation. LangHOPS achieves state-of-the-art results, surpassing previous methods by 5.5% Average Precision (AP) (in-domain) and 4.8% (cross-dataset) on the PartImageNet dataset and by 2.5% mIOU on unseen object parts in ADE20K (zero-shot). Ablation studies further validate the effectiveness of the language-grounded hierarchy and MLLM driven part query refinement strategy. The code will be released here.