You Can Trust Your Clustering Model: A Parameter-free Self-Boosting Plug-in for Deep Clustering

AI summary

Overview

Research area: Deep clustering (unsupervised representation learning) in computer vision, with a focus on post-hoc enhancement of existing clustering models.

Technical level: Advanced. The paper is written for readers comfortable with self-supervised learning, contrastive/self-distillation objectives, EMA teacher-student training, and standard clustering metrics (NMI, ACC, ARI, silhouette coefficient).

Scope in one sentence: The paper introduces DCBoost, a parameter-free plug-in that uses reliable local neighbourhood structure to sharpen the global feature structure of existing deep clustering models.

What This Paper Is About

Deep clustering models have become good at grouping semantically similar samples locally, but the authors observe that their global feature spaces remain entangled — clusters overlap, boundaries are unclear, and intra-class compactness is weak. DCBoost addresses this gap by mining the reliable local cues these models already possess and using them to refine the global structure, without adding any manually tuned hyperparameters. The method is designed as a generic add-on that can be attached to a pre-trained deep clustering model rather than a new model built from scratch.

Key Contributions

1. A new empirical diagnosis. The authors are the first to report that many deep clustering methods simultaneously exhibit poor global structure (low intra-class similarity, noticeable inter-class similarity, low silhouette coefficient) and trustworthy local structure (consistently high k-NN accuracy that often matches or exceeds clustering accuracy).

2. DCBoost, a parameter-free plug-in. The method combines an adaptive k-NN consistency filter for selecting high-confidence samples with a discriminative loss built from three terms — a positive loss, a negative loss, and an instance consistency loss — with no manually tuned hyperparameters.

3. Broad empirical validation. Experiments cover five benchmark datasets and six existing deep clustering models (three representation-based and three clustering-head-based), plus extensions to CLIP-based clustering models.

4. Class-imbalance handling inside the loss. The positive loss includes a weight coefficient $w_c$ derived from the aggregated feature norms of each class, which automatically reduces the influence of classes that contribute disproportionately many high-confidence samples.

Main Findings

• Consistent gains across six models: On five benchmarks, DCBoost improves every integrated model. Table 2 reports average performance moving from 79.2 to 82.2 (+3.0) for ProPos, 72.7 to 77.3 (+4.6) for BYOL, 68.2 to 71.3 (+3.1) for CC, 76.7 to 77.9 (+1.2) for SCAN, 81.1 to 81.9 (+0.8) for CDC, and 79.8 to 80.4 (+0.6) for CoNR.

• Weak baselines benefit most: The paper states that models with lower baseline performance, such as BYOL and CC, gain substantially, while strong models like ProPos still improve by an average of approximately 3.2% in the text (the paper elsewhere describes this as "more than 3% on average").

• Global structure improves dramatically: For ProPos on CIFAR-10, the silhouette coefficient rises from 0.10 to 0.74 and ACC rises from 94.4% to 96.0%. The abstract reports that the silhouette coefficient is amplified by over 7×.

• Local structure is preserved, not degraded: k-NN accuracy remains high and even shows slight improvement after boosting, which the authors interpret as evidence that local reliability survives the refinement.

• Sample selection is essential: Removing the selection mechanism drops average performance from 80.4% to 78.3% when all losses are used, and adding the selection strategy without other changes raises average performance from 75.7% to 78.7%.

• Every loss term matters, and the instance consistency loss is critical: Removing $L_{ins}$ collapses performance on STL-10 (93.6% to 20.5%) and drops CIFAR-10 ACC from 96.0% to 91.1%. Removing $L_{pos}$ yields nearly no improvement, and removing $L_{neg}$ causes a noticeable drop.

• Adaptive k is robust, not just accurate: The adaptive strategy reaches 96.0% ACC on CIFAR-10 versus 95.9% with a manually set k = 30, and 63.9% on CIFAR-20 versus 63.6% with k = 10. The stated advantage is stability rather than raw peak performance.

• Class-balanced weighting helps: On CIFAR-20, the unweighted variant $w_0$ reaches 61.9% ACC, the gradient-detached weighting $w_1$ reaches 63.8%, and the full $w_{ours}$ reaches 63.9%.

• Representation refinement beats head refinement: Applying DCBoost to the representation of CC produces larger ACC gains than applying it to the clustering head; Table 5 lists ACC improvements of 5.0, 8.0, and 2.4 points for the representation and 3.0, 6.1, and 1.0 points for the clustering head across CIFAR-10, CIFAR-20, and ImageNet-Dogs.

• Competitive with a related method (CoNR): On ProPos, CIFAR-10 ACC goes from 94.4 to 95.6 with CoNR and to 96.0 with DCBoost; on CIFAR-20 ACC goes from 61.6 to 63.1 with CoNR and to 63.9 with DCBoost.

• Works on CLIP-based clustering: SIC improves from 92.8 to 92.8 on CIFAR-10 ACC and 57.4 to 58.4 on CIFAR-20; TAC improves from 92.0 to 93.1 on CIFAR-10 and 61.5 to 62.9 on CIFAR-20.

• Beats alternative filtering strategies: Replacing the selection module with MOIT or SSR still helps over baseline, but the full adaptive method performs best; used as complete methods, ProPos+MOIT reaches 94.4% ACC on CIFAR-10 and ProPos+SSR reaches 92.5%, versus 96.0% for ProPos+Ours.

• Scales to a larger benchmark: On Tiny-ImageNet (200 classes, 100,000 samples), the method still improves over ProPos by more than 2%.

Methodology in Plain English

DCBoost starts from any already-trained deep clustering model. It duplicates that model into a "target" network and an "online" network, adds a small randomly initialized predictor, and updates the target network by exponential moving average rather than gradients.

At the end of each training epoch, k-means is run on the target network's features to assign pseudo-labels to all samples. During training, each sample is augmented two ways. The core idea is to only trust samples whose pseudo-label agrees with the pseudo-labels of all of their nearest neighbours in feature space — those are the "high-confidence" anchors used for supervision.

The tricky part is choosing k. Small k accepts many samples but with looser agreement; large k gives cleaner samples but fewer of them. Instead of fixing k, the authors score candidate values from 1 to 50 by multiplying k by the fraction of samples it accepts (geometrically, the area of a rectangle trading off neighbourhood size against sample yield) and pick the best-scoring candidate per batch. This makes the whole procedure free of manually tuned hyperparameters.

Three loss terms then train the network. The positive loss pulls same-pseudo-label high-confidence samples together, weighted by an inverse-norm coefficient that prevents classes with many selected samples from dominating. The negative loss pushes apart the class prototypes computed from high-confidence samples. The instance consistency loss aligns the two augmented views of the same image, with small Gaussian noise (σ = 0.001 in all experiments) added to stabilize training and prevent representation collapse.

Why This Matters

Impact on research. The paper reframes a common failure mode of deep clustering — entangled global structure — as something that can be repaired using that model's own under-used local signal. Because DCBoost is a plug-in rather than a new architecture, it offers a way to compare and improve existing methods on more equal footing, and it introduces silhouette coefficient and k-NN accuracy as a complementary pair of diagnostics for global versus local quality.

Real-world applications:

• Large-scale image organization and automatic album/taxonomy construction where no labels exist.
• Exploratory analysis of scientific or medical imaging archives, where clusters help researchers triage unlabeled scans or specimens.
• Customer or user segmentation from behavioural embeddings in domains without ground-truth labels.
• Pre-annotation and data curation pipelines, where better-separated clusters yield cleaner pseudo-labels for downstream supervised training.
• Retrieval and similarity search, since a better-structured feature space with clearer margins improves nearest-neighbour quality.

Industry relevance. Because the method requires no new hyperparameters and works as a fine-tuning stage on top of a pre-trained model, it is practical for teams that already have a clustering pipeline and want measurable quality gains without redesigning it. Its demonstrated gains on CLIP-based models also matter for organizations building on foundation-model embeddings rather than training encoders from scratch.

Future Directions

1. Scaling to large-scale datasets — the authors explicitly plan to extend DCBoost to much larger benchmarks than the five used here.

2. Removing the known-cluster-count assumption — the conclusion states the method assumes the number of clusters is known, and this is a limitation for real-world scenarios where the class count is unknown.

3. Handling highly imbalanced or non-uniform data distributions — the paper identifies the lack of mechanisms for this as a limitation that can reduce adaptability.

4. More adaptive selection strategies and multimodal extensions — proposed as future work to further boost clustering performance.

Target Audience

Researchers and graduate students working on deep clustering, self-supervised representation learning, and unsupervised computer vision will get the most from this paper, particularly those studying pseudo-label noise filtering or teacher-student training. Practitioners who already have a deep clustering model in production and want a low-friction way to improve cluster separability without hyperparameter tuning are the most likely direct beneficiaries. Readers without a background in contrastive learning or k-NN-based consistency filtering will need to consult the related-work section first.

Authors’ abstract

Recent deep clustering models have produced impressive clustering performance. However, a common issue with existing methods is the disparity between global and local feature structures. While local structures typically show strong consistency and compactness within class samples, global features often present intertwined boundaries and poorly separated clusters. Motivated by this observation, we propose DCBoost, a parameter-free plug-in designed to enhance the global feature structures of current deep clustering models. By harnessing reliable local structural cues, our method aims to elevate clustering performance effectively. Specifically, we first identify high-confidence samples through adaptive $k$-nearest neighbors-based consistency filtering, aiming to select a sufficient number of samples with high label reliability to serve as trustworthy anchors for self-supervision. Subsequently, these samples are utilized to compute a discriminative loss, which promotes both intra-class compactness and inter-class separability, to guide network optimization. Extensive experiments across various benchmark datasets showcase that our DCBoost significantly improves the clustering performance of diverse existing deep clustering models. Notably, our method improves the performance of current state-of-the-art baselines (e.g., ProPos) by more than 3% and amplifies the silhouette coefficient by over $7\times$. Code is available at <https://github.com/l-h-y168/DCBoost>.