EPSegFZ: Efficient Point Cloud Semantic Segmentation for Few- and Zero-Shot Scenarios with Language Guidance

AI summary

Overview

• Research area: 3D computer vision, specifically few-shot and zero-shot semantic segmentation of point clouds, with language-guided multimodal learning.
• Technical level: Advanced (assumes familiarity with attention mechanisms, prototype-based meta-learning, contrastive losses, and point cloud backbones such as DGCNN).
• Scope: The paper introduces EPSegFZ, a pre-training-free point cloud segmentation network that combines a prototype-enhanced attention module, language-guided prototype updates, and a dual relative positional encoding scheme to handle few-shot and zero-shot 3D segmentation.

What This Paper Is About

Few-shot 3D point cloud semantic segmentation (FS-SemSeg) usually requires a two-stage pipeline: a fully-supervised pre-training stage followed by few-shot training, which is resource-intensive and can introduce biases from domain differences between datasets. The authors instead build a network trained entirely from scratch that captures fine-grained high-frequency geometric detail, and they show that exploiting textual annotations from the support set (via CLIP text embeddings) supplies the low-frequency information that would otherwise come from pre-training. The goal is a single efficient model that performs well in both few-shot and zero-shot settings.

Key Contributions

1. EPSegFZ framework: A pre-training-free architecture that the authors report achieves state-of-the-art performance in 3D FS-SemSeg, outperforming the previous state of the art by 5.68% m-IoU on S3DIS and 3.82% m-IoU on ScanNet.
2. Prototype-Enhanced Registers Attention (ProERA): A trainable attention module that subtracts average pooled (low-frequency) features from the attention output to emphasize high-frequency, low-noise information, using learnable register tokens and prototype tokens to reduce the influence of irrelevant background points and mitigate foreground-background imbalance.
3. Language-Guided Prototype Embedding (LGPE): A module that fuses text embeddings from a pre-trained CLIP text encoder into class prototypes with time-varying weights (text weight decays as λ4(t) = λ4*·e^(-0.5t); visual weights grow as λi(t) = λi*·(1 − e^(-0.5t))), reducing reliance on perfect support point clouds and enabling zero-shot inference from text embeddings alone.
4. Dual Relative Positional Encoding (DRPE): The first approach to use query-prototype spatial relationships in the latent space as relative positional encoding for cross-attention in FS-SemSeg, combining sinusoidal encodings of Euclidean distance and cosine angle with no extra training parameters.

Main Findings

• Headline accuracy gains: The method outperforms the previous state of the art by 5.68% and 3.82% m-IoU on the S3DIS and ScanNet benchmarks, respectively.
• S3DIS few-shot results (Ours+DGCNN, mean m-IoU): 73.42 (2-way 1-shot), 76.01 (2-way 5-shot), 65.93 (3-way 1-shot), and 68.28 (3-way 5-shot). The same model with the non-parametric Point-NN backbone reaches 73.26, 75.60, 65.86, and 68.25. For comparison, Seg-PN reaches 66.41 in the 2-way 1-shot setting and SDSimPoint reaches 69.67.
• ScanNet few-shot results (Ours+DGCNN, mean m-IoU): 68.84 (2-way 1-shot), 70.64 (2-way 5-shot), 67.03 (3-way 1-shot), and 69.73 (3-way 5-shot), versus Seg-PN at 63.74 and SDSimPoint at 65.19 in the 2-way 1-shot setting.
• Efficiency: EPSegFZ uses 2.02M parameters, 2.11 GFLOPs, and 0.36 s inference time, compared with COSeg at 7.69M parameters, 9.71 GFLOPs, and 1.35 s, and Seg-PN at 241.67K parameters, 1.95 GFLOPs, and 0.32 s. The reported Δ(m-IoU) relative to the best method in the table is -8.24 for Seg-PN and +0.05 for COSeg.
• Zero-shot feasibility (S3DIS, mean m-IoU): With CLIP embeddings, EPSegFZ reaches 63.84 (2-way 1-shot), 65.43 (2-way 5-shot), 55.62 (3-way 1-shot), and 60.04 (3-way 5-shot), compared with PAPFZS3D using CLIP at 61.09, 64.91, 50.18, and 59.10, and PAPFZS3D using word2vec at 59.98, 63.54, 48.91, and 55.62. 3DGenZ with word2vec is reported at 34.93, 36.12, 23.08, and 27.52.
• Component ablation (S3DIS S0, 2-way 1-shot): The baseline without any component scores 31.55. Adding only ProERA gives 64.84, only LGPE gives 60.22, only DRPE gives 59.27; ProERA+LGPE gives 70.48, ProERA+DRPE gives 69.35, LGPE+DRPE gives 70.17, and all three together give 73.08.
• Prototype ablation (S3DIS S0, 2-way 1-shot): Using only the text prototype yields 68.71, only the dynamic prototype yields 71.49, and only the raw prototype yields 70.30; combining all three prototypes reaches 73.08. The authors state that dynamic prototypes have the most significant impact on performance.
• Loss ablation: Removing the alignment loss gives 69.46 and removing the contrastive loss gives 69.30; removing both drops performance to 68.55, versus 73.08 with the full loss.
• Positional encoding comparison: DRPE achieves 73.08, compared with 71.22 for a learnable positional encoding and 69.94 for a sinusoidal positional encoding.
• Qualitative observations: Visualized frequency spectra show EPSegFZ's prototype features are richer and more uniform across frequency bands than Seg-PN's, which predominantly capture low-frequency information. t-SNE visualizations show same-class features form more compact clusters than PAPFZS3D or Seg-PN, and segmentation visualizations show clearer edge delineation on S3DIS.
• Register and hyperparameter tuning: N+1 registers work best for an N-way K-shot task; ablation on prototype and decoder block numbers indicates 100 prototypes and 3 decoder blocks give the best performance-efficiency trade-off.
• Training behavior: Loss curves over the first 5,000 iterations show EPSegFZ starts with a higher loss over the first 3,000 iterations due to its randomly initialized backbone but converges to stable values comparable to a fully-supervised pre-trained method.

Methodology in Plain English

For each training episode, support and query point clouds are mapped into a latent space by a DGCNN backbone trained from scratch (a non-parametric Point-NN backbone is also tested). Multi-Prototype Sampling derives candidate prototypes from the support features.

The ProERA module then appends learnable register tokens and prototype tokens to the query and prototype features and runs self-attention over them. Because self-attention behaves like a low-pass filter, the module subtracts the average of its input features from the attention output, leaving high-frequency-dominant features that preserve object edges. The registers also spread attention across the scene, so distinct regions (background versus object-dense areas) are emphasized differently.

The LGPE module then refines the prototypes by blending four sources: the previous prototype token, the raw fine-grained prototype, a mask-average-pooled dynamic prototype, and a text prototype from CLIP prompted with the support class names. The blend weights shift over training so that text dominates early (when the randomly initialized visual features are weak) and learned visual features dominate later.

The DRPE module computes, for every query point and each prototype, both the Euclidean distance and the cosine angle between their feature vectors. Both quantities are converted with a sinusoidal positional encoding function and added together, and the result is fed into cross-attention (without introducing trainable parameters) so that query points are matched to the correct prototype. Final predictions come from a softmax over the dot product between normalized query features and normalized prototypes.

Training uses three losses: a cross-entropy segmentation loss, an InfoNCE-based foreground-consistency contrastive loss, and a foreground-aware alignment loss that minimizes cross-entropy between text-visual similarities and class labels to align the two modalities.

Experiments are run on the S3DIS and ScanNet datasets, split into disjoint S0 and S1 subsets, with 2048 points sampled per unit, 30,000 training iterations, and 100 randomly selected episodes at inference. Text embeddings are precomputed and stored locally as a dictionary. Training used two Nvidia RTX 3090 GPUs with 24GB memory (testing on a single RTX 3090), with PyTorch 2.2.0 and CUDA 12.1.

Why This Matters

• Research impact: The work challenges the assumption that strong 3D few-shot segmentation requires a fully-supervised pre-training stage, and it demonstrates that textual annotations are an underused resource in FS-SemSeg settings where labeled point clouds are scarce.
• Real-world applications:
  • Warehouse and logistics robots that must recognize new object categories from only a few demonstrations.
  • Autonomous vehicles and mobile robots segmenting unfamiliar scene elements in novel environments without retraining.
  • Augmented reality and mixed-reality systems that need to label 3D scene geometry with minimal user annotation.
  • Industrial inspection and manufacturing robotics, where object categories may be introduced frequently and labeled data is costly.
• Industry relevance: The efficiency profile (2.02M parameters, low FLOPs, sub-second inference) is aimed at lightweight and embedded deployment, where the hardware cannot support large pre-trained 3D backbones. The authors' acknowledgement lists support from the National Robotics Programme's "Domain-specific Robotics Foundation Models for Manufacturing (DS-RFM)" initiative and the Centre for Advanced Robotics Technology Innovation, and notes discussions with Hexagon's Simufact Engineering GmbH.

Future Directions

• Whether the language-guided prototype scheme can be extended to large-scale 3D foundation models; the authors explicitly decline to compare against models such as SegPoint because of differences in training resources and data scale.
• How to further bridge the gap between the DGCNN backbone (reported at 68.84 mean m-IoU on ScanNet 2-way 1-shot) and the Point-NN variant (reported at 65.86), since the authors test both but do not resolve which design should be preferred in practice.
• Whether register count and prototype count (found best at N+1 registers, 100 prototypes, 3 decoder blocks) scale predictably to larger numbers of classes or higher N-way tasks.
• Whether improved zero-shot performance depends primarily on the choice of language model, given that swapping word2vec for CLIP changed PAPFZS3D's S3DIS 2-way 1-shot score from 59.98 to 61.09.

Target Audience

Researchers and graduate students working on 3D point cloud understanding, few-shot or meta-learning, and vision-language multimodal models. It is also relevant to applied engineers building lightweight perception systems for robotics and manufacturing who need models that adapt to new object categories without a costly pre-training stage. Readers should already be comfortable with attention mechanisms, prototype-based segmentation, and standard 3D segmentation benchmarks to follow the methodology and ablation tables.

Authors’ abstract

Recent approaches for few-shot 3D point cloud semantic segmentation typically require a two-stage learning process, i.e., a pre-training stage followed by a few-shot training stage. While effective, these methods face overreliance on pre-training, which hinders model flexibility and adaptability. Some models tried to avoid pre-training yet failed to capture ample information. In addition, current approaches focus on visual information in the support set and neglect or do not fully exploit other useful data, such as textual annotations. This inadequate utilization of support information impairs the performance of the model and restricts its zero-shot ability. To address these limitations, we present a novel pre-training-free network, named Efficient Point Cloud Semantic Segmentation for Few- and Zero-shot scenarios. Our EPSegFZ incorporates three key components. A Prototype-Enhanced Registers Attention (ProERA) module and a Dual Relative Positional Encoding (DRPE)-based cross-attention mechanism for improved feature extraction and accurate query-prototype correspondence construction without pre-training. A Language-Guided Prototype Embedding (LGPE) module that effectively leverages textual information from the support set to improve few-shot performance and enable zero-shot inference. Extensive experiments show that our method outperforms the state-of-the-art method by 5.68% and 3.82% on the S3DIS and ScanNet benchmarks, respectively.