SSR: Semantic and Spatial Rectification for CLIP-based Weakly Supervised Segmentation

AI summary

Overview

• Research area: Computer vision, specifically Weakly Supervised Semantic Segmentation (WSSS) using CLIP-based vision-language models.
• Technical level: Advanced. The paper assumes familiarity with CLIP, Vision Transformers, multi-head self-attention, contrastive learning, superpixel segmentation, and pseudo-label pipelines.
• Scope: The paper introduces a two-part rectification framework (SSR) that corrects over-activation in non-target foreground regions and background regions of CLIP-generated class activation maps, evaluated on PASCAL VOC 2012 and MS COCO 2014.

What This Paper Is About

Weakly supervised segmentation tries to produce pixel-level masks using only cheap image-level labels, typically by generating Class Activation Maps (CAM) and refining them into pseudo-labels. CLIP-based methods produce strong initial CAMs, but they still misactivate the wrong foreground objects and background areas because of a semantic gap between image and text representations (the "modality gap") and because noisy affinity values propagate background information during refinement. The paper's goal is to fix both failure modes: semantic misalignment at the feature level and spurious background responses at the spatial level.

Key Contributions

1. A Semantic and Spatial Rectification (SSR) method that targets the two over-activation problems in CLIP-based WSSS: erroneous activation of non-target foreground regions and of background areas.
2. Cross-Modal Prototype Alignment (CMPA), a semantic-level contrastive learning mechanism that aligns image and text feature representations in a shared embedding space, reducing inter-class overlap while strengthening semantic correlations. It uses structurally identical but parameter-independent Image Semantic Alignment (ISA) and Text Semantic Alignment (TSA) modules.
3. Superpixel-Guided Correction (SGC), a spatial-level module that uses SLIC superpixel segmentation with K-means clustering of superpixel regions to build a binary mask, which then filters non-target columns from a fused CLIP/DINO affinity matrix before CAM refinement.
4. Extensive experiments on PASCAL VOC and MS COCO reporting state-of-the-art results among both single-stage and multi-stage approaches.

Main Findings

• Segmentation accuracy: SSR reaches 79.5% mIoU on PASCAL VOC val, 79.6% on VOC test, and 50.6% on MS COCO val. Without CRF post-processing the numbers are 78.2%, 78.1%, and 49.2% respectively. The paper states this surpasses multi-stage methods by up to 0.6% and CLIP-based ExCEL by 0.3%.
• CAM seed quality: On the VOC training set, SSR improves CAM quality to 78.7%, which the paper reports as surpassing the prior state of the art by at least 0.7%.
• Component ablation (SGC): The CMPA-generated initial CAM achieves 63.3% mIoU; adding CLIP's multi-head attention raises it to 74.6%; further adding DINO's attention raises it to 76.3%; the complete SGC module reaches 78.7% mIoU on VOC train.
• Loss function ablation: A CLIP baseline achieves 58.6% mIoU. Direct feature fine-tuning (L_feature) degrades performance by 5.1% to 53.5%. Intra-modal contrastive learning (L_in_modal) drops marginally by 0.8% to 57.8%. The proposed cross-modal contrastive loss (L_cross_modal) improves mIoU by 4.7% to 63.3%, which the authors attribute to aligning same-class cross-modal representations while separating different classes.
• Multi-metric comparison (VOC val): SSR scores 0.795 mIoU, 0.879 precision, 0.891 recall, and 0.198 confusion ratio, versus MoRe at 0.764 / 0.837 / 0.847 / 0.239, WeCLIP at 0.764 / 0.844 / 0.861 / 0.237, and SeCo at 0.740 / 0.84 / 0.849 / 0.232. The paper reports this as beating the best baseline MoRe by 3.1% in mIoU, 3.5% in precision, 3% in recall, and a 3.4% reduction in confusion ratio.
• Comparison to fully supervised: On VOC val, SSR reaches 79.5% mIoU, which is stated as 97.4% of fully supervised performance. Fully supervised reference points listed are DeepLabV2 with RN101 at 77.7% and DeepLabV2 with ViT-B at 82.3%.
• Feature alignment visualization: t-SNE visualizations on PASCAL VOC 2012 validation images show the original CLIP feature space as sparse with class overlap, while CMPA produces more compact intra-class clusters and clearer inter-class boundaries (the paper cites dining table/bottle separation as an example).
• Qualitative behavior: CAM visualizations show CMPA initial CAMs are more complete than CLIP's; the paper gives the example that CLIP erroneously activates adjacent "chair" regions for the "sofa" category, and this error is amplified downstream. After SGC refinement, background activations are visibly suppressed.

Methodology in Plain English

The method starts from a CLIP model that compares images and text in a shared embedding space, plus a DINO model whose self-supervised attention captures finer spatial structure.

At the semantic level, rather than only rewriting the text prompts—which the authors argue leaves the underlying cross-modal gap intact—SSR passes image and text features through two parallel modules (ISA and TSA) that project them into a common space. For each image, class activation maps are produced with GradCAM and used to compute image features (via masked average pooling over the activation maps) and text features (selected by class index). These foreground-aware features from across the dataset are clustered with K-means to build image prototypes and text prototypes. Contrastive learning then pulls each image feature toward text prototypes of the same class, pulls text prototypes toward visual prototypes of the same class, and pushes apart prototypes of different classes, using cluster-derived pseudo-labels to define positives and negatives. The temperature in this contrastive term is a learnable parameter.

At the spatial level, SSR segments each input image with the SLIC superpixel algorithm, clusters the superpixel regions by color-space information with K-means, and keeps only regions whose ratio of high-confidence activation to total activation exceeds a threshold. Those regions form a binary mask. Meanwhile, the affinity matrix is built by concatenating and normalizing the multi-head self-attention features from CLIP and DINO (CLIP for high-level semantics, DINO for fine-grained spatial relations), with a CLIP:DINO weighting ratio of 0.4:0.6. The binary mask is applied element-wise to this affinity matrix to delete non-target columns, and the resulting corrected affinity matrix is used to propagate and refine the initial CAM. The authors note SLIC was chosen over heavier models like SAM because of its lightweight design.

Training combines the prototype contrastive loss with a segmentation loss on online-generated pseudo masks, weighted by a factor of 0.1. The architecture uses CLIP ViT-B/16 and DINO ViT-S/16 (DINOv1), optimized with AdamW at a learning rate of 1e-5 and weight decay of 2e-3; prototypes are refreshed every 5,000 iterations. PASCAL VOC uses batch size 128 and 30,000 iterations, while MS COCO uses batch size 256 and 80,000 iterations. The prototype temperature coefficient is 0.05.

Why This Matters

The work argues that prior CLIP-based WSSS improvements focus on the text side alone and therefore cannot close the underlying cross-modal representation gap. By correcting both the semantic space and the spatial affinity matrix, SSR raises weakly supervised segmentation closer to what fully supervised models achieve, while keeping annotation costs low. This matters because pixel-level labeling is the single most expensive part of building segmentation systems.

Real-world applications:

• Medical imaging, where voxel- or pixel-level expert annotation is scarce and expensive but image-level labels are more available.
• Autonomous driving and robotics, where scene segmentation models must be retrained on new environments and object categories.
• Remote sensing and satellite imagery analysis at scale, where per-pixel labeling is impractical.
• Industrial inspection and content creation tools (photo/video editing and background removal), where users can label what is in an image but not where each object lies.

Industry relevance: methods that convert image-level tags into usable segmentation masks can dramatically lower dataset construction costs for companies deploying perception systems. The paper also emphasizes practical deployment properties—SLIC is chosen specifically because it is lighter than alternatives like SAM, and the reported mIoU without CRF (78.2% on VOC val) shows the pipeline retains most of its accuracy without a costly post-processing step.

Future Directions

• Closing the remaining gap to fully supervised segmentation, since SSR reaches 97.4% of fully supervised performance on VOC val but does not match the 82.3% DeepLabV2 ViT-B reference.
• Extending the rectification framework to the other weak annotation types the paper lists in its introduction (points, scribbles, and bounding boxes), where the noise characteristics differ from image-level labels.
• Investigating whether the reliance on a separate DINO attention branch can be reduced or replaced, since the ablation shows DINO attention contributes an additional 1.7% mIoU (76.3% to 78.7%) and adds a second backbone to the pipeline; sensitivity to the CLIP:DINO weighting and the superpixel activation-ratio threshold is also not reported in the provided content.
• Testing transferability beyond the two evaluated benchmarks, since the paper does not report results on datasets outside PASCAL VOC 2012 and MS COCO 2014, nor on cross-dataset or domain-shift settings.

Target Audience

• Researchers working on weakly supervised semantic segmentation, class activation maps, and pseudo-label generation.
• Practitioners applying CLIP or other vision-language models to dense prediction tasks.
• Engineers who need low-annotation-cost segmentation pipelines for medical imaging, remote sensing, autonomous driving, or industrial inspection.
• Graduate students and advanced undergraduates with background in deep learning, Vision Transformers, and contrastive learning who want a concrete example of diagnosing and fixing cross-modal alignment problems.

Authors’ abstract

In recent years, Contrastive Language-Image Pretraining (CLIP) has been widely applied to Weakly Supervised Semantic Segmentation (WSSS) tasks due to its powerful cross-modal semantic understanding capabilities. This paper proposes a novel Semantic and Spatial Rectification (SSR) method to address the limitations of existing CLIP-based weakly supervised semantic segmentation approaches: over-activation in non-target foreground regions and background areas. Specifically, at the semantic level, the Cross-Modal Prototype Alignment (CMPA) establishes a contrastive learning mechanism to enforce feature space alignment across modalities, reducing inter-class overlap while enhancing semantic correlations, to rectify over-activation in non-target foreground regions effectively; at the spatial level, the Superpixel-Guided Correction (SGC) leverages superpixel-based spatial priors to precisely filter out interference from non-target regions during affinity propagation, significantly rectifying background over-activation. Extensive experiments on the PASCAL VOC and MS COCO datasets demonstrate that our method outperforms all single-stage approaches, as well as more complex multi-stage approaches, achieving mIoU scores of 79.5% and 50.6%, respectively.