Novel Class Discovery for Point Cloud Segmentation via Joint Learning of Causal Representation and Reasoning

AI summary

Overview

• Research area: 3D computer vision, specifically Novel Class Discovery for point cloud semantic segmentation (3D-NCD), approached through causal representation learning and causal reasoning.
• Technical level: Advanced.
• Scope: The paper reformalizes 3D-NCD with a structural causal model (SCM) and proposes a joint learning framework that removes shortcut (confounding) features from base class representations and models base-to-novel causal relations with a graph, evaluated on two 3D datasets and extended to two 2D datasets.

What This Paper Is About

Standard point cloud segmentation assumes a "closed world" where every class is labeled during training. This paper targets the harder 3D-NCD setting, where a model must segment unlabeled novel 3D classes using supervision only from labeled base classes. The authors argue that existing methods learn statistical shortcuts (confounding features) rather than causal class properties, which causes novel classes to be misclassified as base classes — for example, a novel "chair" being labeled "stool" because both share a visually prominent "circular support," while the actual causal feature is the "leg."

Key Contributions

1. Introducing causality to 3D-NCD. The authors state this is the first method to incorporate causality into 3D Novel Class Discovery, using a structural causal model with four variables — base class $B$, novel class $N$, original point cloud $X$, and confounding factor $U$ — to diagnose why statistical correlation learning confuses novel classes.
2. Causal representation prototype learning (CRP). A causal adversarial mechanism forces the learned base-class feature representation $Z$ to be independent of the confounding factor $U$ (minimizing mutual information $I(Z;U)$), and the deconfounded features are then aggregated into base class causal prototypes.
3. Graph-based causal reasoning (CRG). A graph whose nodes are base class causal prototypes and novel class prototypes is built with a causal adaptive adjacency matrix, together with two constraints: causal pruning (removing low-weight edges via a learnable threshold $\theta$) and inference direction consistency (enforcing that information flows along the true causal direction).
4. GCN-based pseudo-label generation (GCPL). The optimized graph is fed into a graph convolutional network to produce base class labels and novel class pseudo-labels, replacing direct similarity matching between novel and base classes.

Main Findings

• SemanticPOSS results: The method achieves the highest mIoU across all classes in all splits. In split 2, base class mIoU reaches 56.1%, a 3.3% improvement over DASL. In split 3, novel class performance exceeds DASL by 10.2%, including 36.2% IoU on the cone class, where DASL and NOPS both achieve 0.
• SemanticKITTI results: The method achieves the highest mIoU across all classes in all splits. Novel class segmentation improves by 4.9% on split 1, where the car class reaches 82.6% IoU, a 13.9% improvement over SNOPS. On the motorcycle class of split 3, the method achieves an 11.5% improvement.
• Ablation progression: Starting from a modified MinkowskiUNet-34C baseline (Building 58.6, Car 6.0, Ground 44.9, Plants 43.7, split-0 average 38.3, overall 24.7), adding causal representation prototypes raises the split-0 average to 41.6 and overall to 25.9; further stages raise these to 45.5 and 28.8, then 47.4 and 30.1, with the full pipeline reaching 60.4, 8.3, 80.8, 55.3 (split-0 average 51.2) and 32.5 overall.
• 2D extension on PASCAL-5^i: 71.9, 62.5, 57.3, 53.1 across Fold0–Fold3, averaging 61.2, versus EUMS at 69.8, 60.1, 56.3, 50.2, averaging 59.1.
• 2D extension on COCO-20^i: 43.23, 25.91, 20.30, 18.56, averaging 27.00, versus EUMS at 42.39, 26.89, 19.75, 18.19, averaging 26.81. The paper notes the method outperforms EUMS in most folds; Fold1 is the exception.
• Visual evidence: Grad-CAM visualizations on PASCAL-5^i and feature map visualizations on COCO-20^i show more focused and clearer causal activation regions than EUMS, which produces scattered and dispersed activations. On 3D data, NOPS and DASL confuse "Plants" with "Building," "Road" with "Sidewalk," and "Terrain" with "Vegetation," while the proposed method separates them.
• Reported scope limit: Because novel classes have no labels and pseudo-labels may carry bias or noise, the paper deliberately extracts causal representations only for base classes and uses them as prior knowledge for novel class inference.

Methodology in Plain English

The authors start by drawing a causal diagram of the task: the original point cloud $X$ produces both base class $B$ and novel class $N$; the base class causally influences the novel class ($B \to N$); and an unobserved confounding factor $U$ leaks into the base class representation ($U \to B$) as shortcut features. Since a perfect statistical adjustment for $U$ is impractical — the authors note that prior 2D methods approximate it with an average of visual features — they instead train adversarially: a feature extractor tries to classify base points correctly while an adversarial network tries to recover $U$ from the extracted features. If the adversarial network fails, the features are considered deconfounded. Base class prototypes are then updated as weighted averages of the deconfounded point features.

Next, a graph is constructed with base class causal prototypes and novel class prototypes as nodes, and edge weights computed by an attention mechanism with a temperature coefficient. Two losses shape the graph: a direction loss that penalizes propagation against the causal direction, and a pruning loss that removes edges whose weights fall below a learnable threshold. A graph convolutional network then propagates information across the graph, and novel class prototypes are matched to the nearest base class prototype by cosine similarity to assign pseudo-labels. In the ablation, the authors also show a variant that generates novel class pseudo-labels with Sinkhorn-Knopp, consistent with existing methods.

Implementation uses MinkowskiUNet-34C as the backbone, AdamW with an initial learning rate of 1e-3 decaying every 5 epochs to a minimum of 1e-5, $\lambda_{adv}$ and $\theta$ both initialized to 0.5 and adjusted dynamically, temperature $\tau$ of 0.06, regularization coefficient $\lambda$ of 0.02, and 3 graph convolution layers. Evaluation uses SemanticKITTI (sequence 08) and SemanticPOSS (sequence 03), with the NOPS partitioning strategy; base class IoU is reported directly, and novel class clusters are matched to ground truth with the Hungarian algorithm.

Why This Matters

• Research impact: The paper reframes 3D-NCD as a causal inference problem rather than a similarity-matching problem, and shows that the same causal representation-and-reasoning recipe also transfers to 2D NCD segmentation on PASCAL-5^i and COCO-20^i. It also distinguishes its confounding factor from prior 3D causal work: whereas CausalPC treats adversarial perturbations and sensor noise as $U$, here $U$ denotes non-causal shortcut features.
• Real-world applications (as motivated by the paper):
  • Autonomous driving, where new object categories may appear on the road without prior definitions or labels.
  • Domestic robotics, where household objects change and cannot be exhaustively pre-labeled.
  • Robotic perception systems generally, which need to operate under a dynamic open-world assumption.
  • LiDAR-based scene understanding pipelines that must scale annotation-free to unseen classes.
• Industry relevance: Because 3D-NCD requires neither labeled novel class samples (as in few-shot 3D segmentation) nor textual or external semantic guidance (as in zero-shot and open-vocabulary 3D segmentation), the setting is the least constrained of these alternatives and therefore attractive for deployment where labeling budgets and predefined class vocabularies are limited.

Future Directions

• Extending causal representation learning to novel classes. The paper explicitly limits its causal representation extraction to base classes, stating that pseudo-label-derived features for novel classes cannot be guaranteed consistent with true causal relationships; extending deconfounding to novel classes is the obvious open problem.
• Reducing the gap to fully supervised performance. On SemanticPOSS the fully supervised "Full" reference reports an all-class value of 51.5, while the best reported full-method all-class value for split 0 is 41.7 and for split 3 is 47.2; on SemanticKITTI the Full all-class reference is 50.3.
• Handling dynamic and streaming novel classes. The paper notes that causal relationships change as novel class nodes are introduced and designs a dynamically adjusted threshold, but catastrophic forgetting in an open world is named as a motivation rather than fully evaluated.
• Understanding the uneven 2D gains. On COCO-20^i Fold1, EUMS scores 26.89 against the proposed method's 25.91; the paper reports overall superiority in "most fold" but does not analyze why this fold differs.
• Broader benchmarking. The paper reports experiments only on SemanticKITTI and SemanticPOSS for 3D and PASCAL-5^i and COCO-20^i for 2D; generalization to other backbones and datasets is not reported.

Target Audience

Researchers and graduate students working on 3D point cloud segmentation, novel class discovery, open-world or open-vocabulary perception, and causal representation learning. It is also relevant to practitioners building LiDAR-based perception for autonomous driving or robotics who need models that can handle unseen object categories without additional labels or text descriptions. Readers should be comfortable with causal graphical models, mutual information minimization, prototype learning, and graph neural networks, as the paper assumes this background without extended exposition.

Authors’ abstract

In this paper, we focus on Novel Class Discovery for Point Cloud Segmentation (3D-NCD), aiming to learn a model that can segment unlabeled (novel) 3D classes using only the supervision from labeled (base) 3D classes. The key to this task is to setup the exact correlations between the point representations and their base class labels, as well as the representation correlations between the points from base and novel classes. A coarse or statistical correlation learning may lead to the confusion in novel class inference. lf we impose a causal relationship as a strong correlated constraint upon the learning process, the essential point cloud representations that accurately correspond to the classes should be uncovered. To this end, we introduce a structural causal model (SCM) to re-formalize the 3D-NCD problem and propose a new method, i.e., Joint Learning of Causal Representation and Reasoning. Specifically, we first analyze hidden confounders in the base class representations and the causal relationships between the base and novel classes through SCM. We devise a causal representation prototype that eliminates confounders to capture the causal representations of base classes. A graph structure is then used to model the causal relationships between the base classes' causal representation prototypes and the novel class prototypes, enabling causal reasoning from base to novel classes. Extensive experiments and visualization results on 3D and 2D NCD semantic segmentation demonstrate the superiorities of our method.