Hierarchical Semantic Alignment for Image Clustering

AI summary

Overview

Research area: Computer vision, specifically unsupervised image clustering that uses external textual knowledge from vision-language models (CLIP), WordNet nouns, and image captions.

Technical level: Advanced. The paper assumes familiarity with CLIP embeddings, k-means clustering, optimal transport, Sinkhorn-Knopp iterations, softmax attention weighting, and standard clustering metrics (NMI, ACC, ARI).

Scope: The paper proposes CAE (hierarChical semAntic alignmEnt), a training-free pipeline that combines noun-level and caption-level text semantics with CLIP image features and fuses them adaptively before k-means clustering, and it evaluates this pipeline on eight datasets.

What This Paper Is About

Recent image clustering methods improve results by pulling in external text knowledge, usually nouns retrieved from WordNet, to sharpen the boundaries between visually similar clusters. The authors argue that nouns carry inherent ambiguity: a single noun can name entirely different objects ("crane" as a bird or a machine), and semantically similar nouns ("spaniel", "saluki", "truck", "tractor") fail to separate fine-grained classes. The goal is to build a clustering method that keeps the categorical information nouns provide while adding the fine-grained attribute detail found in image captions, without training any parameters.

Key Contributions

1. A training-free clustering method, CAE, that exploits external semantic knowledge from two complementary sources at once: nouns from WordNet and image descriptions from caption datasets.
2. A semantic space construction procedure that aligns image feature distributions with the selected relevant nouns and descriptions, using optimal transport to compute a noun counterpart and a caption counterpart for every image.
3. An adaptive semantics fusion strategy that weights image, noun, and caption features per instance through a prototype-guided, temperature-scaled softmax, producing a fused embedding for clustering.
4. Extensive experiments on five classic datasets (STL-10, CIFAR-10, CIFAR-20, ImageNet-10, ImageNet-Dogs) and three challenging datasets (DTD, UCF-101, ImageNet-1K), showing consistent gains over baselines including training-based methods, the training-free TAC method, and zero-shot CLIP.

Main Findings

• ImageNet-1K headline gain: CAE surpasses the state-of-the-art training-free approach by 4.2% in accuracy and 2.9% in adjusted rand index on ImageNet-1K (reported in the abstract and matching Table 2, where CAE reaches 53.1 ACC / 39.3 ARI against TAC's 48.9 ACC / 36.4 ARI).
• Classic dataset gains over TAC: CAE achieves a 7.2% improvement in ARI on STL-10 (96.7 versus TAC's 89.5) and a 5.1% accuracy improvement on CIFAR-20 (60.9 versus 55.8).
• Best average on classic benchmarks: Across STL-10, CIFAR-10, CIFAR-20, ImageNet-10 and ImageNet-Dogs, CAE averages the highest score in the training-free group (81.9 average, versus 79.5 for TAC, 78.7 for zero-shot CLIP and 66.3 for CLIP k-means) and also exceeds the best training-based baseline TCL (71.4).
• One exception: On CIFAR-10, the training-based SIC method achieves the best performance; CAE is second-best there (90.9 ACC versus SIC's 92.6 ACC).
• Challenging datasets: CAE obtains the best average across DTD, UCF-101 and ImageNet-1K (57.0, versus 56.7 for zero-shot CLIP, 54.8 for TAC and 54.6 for SIC), and it outperforms zero-shot CLIP on DTD and UCF-101 even though zero-shot CLIP has access to candidate class names that CAE does not use.
• Nouns and captions are complementary: Comparing the two similarity scores in Figure 2, image embeddings of a robin and a linnet sit at 0.73 cosine similarity; nouns alone lower it to 0.64, captions alone to 0.56, and the combination to 0.35.
• Combining the two semantics beats either alone: In the ablation, using only noun counterparts or only caption counterparts yields similar results, combining both improves them, and the full fused representation is best across all metrics and datasets — for example a 2.6% NMI, 2.3% ACC and 2.4% ARI improvement on ImageNet-Dogs over the noun-only row.
• Adaptive fusion beats concatenation and summation: Concatenating image, noun and caption features gives the lowest average (53.6), simple summation is better (55.6), and the prototype-weighted adaptive fusion is best (57.0). The paper reports a 3.9% ACC improvement attributed to the adaptive fusion operation.
• Top-K selection matters: Varying the number of selected nouns and descriptions from 1 to 30, a top-K of 1 captures too little semantics, moderate values improve NMI and ACC, and further increases yield diminishing returns or slight degradation from noise.
• Optimal transport is theoretically preferable to softmax weighting: Theorem 1 states that softmax aggregation is a special case of entropic optimal transport with the column-wise marginal constraint relaxed, that softmax generally violates the balance constraint over noun embeddings, and that softmax incurs higher semantic error than optimal transport with probability at least 1 − δ.
• Visualization: t-SNE plots on ImageNet-Dogs show CLIP image embeddings only partially separated, noun and caption embeddings better separated, and the fused CAE embeddings the most clearly separated with minimal overlap.

Methodology in Plain English

The pipeline runs in two stages and trains nothing.

First, the method builds a text-side semantic space that matches the dataset. It runs k-means on the CLIP image embeddings and, following the setting in prior work, splits the data into n = N/300 groups. For each group it computes a semantic center by averaging the image embeddings assigned to it, then scores every noun in WordNet and every caption from Flickr by how probably it belongs to each center, and keeps the top-K nouns and top-K captions for each cluster. The union of those selections becomes the working vocabulary and the working caption pool. Because the clusters are only a coarse partition, this step finds text that is broadly relevant without needing any true labels.

Second, for each image the method asks: which of the selected nouns (and which of the selected captions) best describe this specific image? Rather than a simple similarity-weighted average, it frames the match as an optimal transport problem between the distribution of image embeddings and the distribution of selected text embeddings, using cosine distance as the transport cost and the Sinkhorn-Knopp algorithm to approximate the plan. The transport plan plus the similarity scores combine into a noun counterpart vector and a caption counterpart vector for each image. Optimal transport is used because it forces the text vocabulary to be used in a balanced way across images, whereas plain softmax weighting lets every image grab the same few text items.

Third, the image, noun and caption vectors are fused per instance. The method averages all three into a semantic prototype, measures the cosine similarity of each modality to that prototype, turns those similarities into weights with a temperature-scaled softmax (temperature 0.01), and takes a weighted sum. Modalities that agree more with the prototype get more weight. Finally, standard k-means is run on the fused embeddings to produce cluster assignments.

Experiments use CLIP with ViT-B/32 as the image encoder and a Transformer as the text encoder, nouns placed in the template "a photo of [CLASS]", and all runs on a single Nvidia RTX 3090 GPU.

Why This Matters

Impact on research. The paper pushes the externally-guided clustering paradigm forward by showing that the ambiguity of a single knowledge source (nouns) can be corrected with a second, differently granular source (captions). It also provides a theoretical argument for using optimal transport instead of softmax similarity weighting, and it demonstrates that a fully unsupervised clustering pipeline can beat zero-shot CLIP on some datasets even without access to the class names that zero-shot CLIP relies on.

Potential real-world applications:

• Photo and media library organization: grouping large unlabeled image collections by semantic content without any human annotation or class-name list.
• E-commerce and marketplace cataloging: sorting product images into categories and subcategories when the taxonomy is unknown or when fine-grained distinctions (for example, similar vehicle or pet categories) matter.
• Scientific and biodiversity monitoring: separating visually similar species or specimen images where coarse category names alone cannot distinguish classes.
• Medical and industrial image triage: clustering unlabeled scans, defect images or sensor imagery to surface natural groupings for later expert review.

Industry relevance. The method is training-free and runs on a single consumer-grade GPU (RTX 3090), which lowers the cost of deploying clustering on proprietary image collections. Because it needs no class-name priors and no paired image-text data, it fits scenarios where organizations hold large unlabeled image sets but cannot afford annotation, and it can serve as a preprocessing step for labeling, retrieval, or dataset curation pipelines.

Future Directions

• Obtaining more precise image descriptions by leveraging multimodal large language models, which the authors name explicitly as future work.
• Better handling of the balance between semantic richness and redundancy, since increasing the number of selected nouns and descriptions beyond a moderate point produces diminishing or slightly negative returns.
• Investigating the trade-off between noun-level and caption-level knowledge further, given that the paper reports the two are complementary but does not report a systematic study of which source dominates under which dataset conditions.
• Extending the optimal transport alignment beyond the marginal-balance argument in Theorem 1, for example by testing alternative transport costs or regularization settings, since the theorem only establishes that softmax is a relaxed special case.

Target Audience

Researchers and graduate students working on unsupervised clustering, vision-language models, and cross-modal representation learning; practitioners who need to group unlabeled image collections without training a model or supplying category names; and readers interested in how external lexical resources such as WordNet can be combined with caption data to compensate for the ambiguity of any single text source.

Authors’ abstract

Image clustering is a classic problem in computer vision, which categorizes images into different groups. Recent studies utilize nouns as external semantic knowledge to improve clustering performance. However, these methods often overlook the inherent ambiguity of nouns, which can distort semantic representations and degrade clustering quality. To address this issue, we propose a hierarChical semAntic alignmEnt method for image clustering, dubbed CAE, which improves clustering performance in a training-free manner. In our approach, we incorporate two complementary types of textual semantics: caption-level descriptions, which convey fine-grained attributes of image content, and noun-level concepts, which represent high-level object categories. We first select relevant nouns from WordNet and descriptions from caption datasets to construct a semantic space aligned with image features. Then, we align image features with selected nouns and captions via optimal transport to obtain a more discriminative semantic space. Finally, we combine the enhanced semantic and image features to perform clustering. Extensive experiments across 8 datasets demonstrate the effectiveness of our method, notably surpassing the state-of-the-art training-free approach with a 4.2% improvement in accuracy and a 2.9% improvement in adjusted rand index (ARI) on the ImageNet-1K dataset.