Multi-modal Dynamic Proxy Learning for Personalized Multiple Clustering

AI summary

Overview

• Research area: Unsupervised machine learning, specifically multi-modal (vision plus language) multiple clustering, where a system must produce several different valid groupings of the same dataset according to different user-specified interests.
• Technical level: Advanced. The paper assumes familiarity with contrastive vision-language pretraining (CLIP), attention mechanisms, contrastive losses, and clustering evaluation metrics such as NMI and Rand index.
• Scope: The paper proposes Multi-DProxy, a framework that replaces fixed textual prompt lists with learnable, dynamically refined "proxy" embeddings, fused with visual features through a gated attention module, and evaluates it on seven image datasets against eight multiple-clustering baselines (arXiv:2511.07274v1 [cs.LG], 10 Nov 2025).

What This Paper Is About

Existing multiple-clustering methods produce many possible partitions of a dataset but do not know which ones a user actually cares about, so a user must manually sift through the output. Recent multi-modal approaches use CLIP with predefined candidate word lists (for example, "red", "blue", "green" for the interest "color") to steer clustering toward user intent, but these lists are static and the way vision and text are combined is fixed.

The goal of this paper is to make both parts adaptive: textual proxies become learnable and are iteratively refined against emerging cluster structure, while visual and textual features are combined through a gated, learnable attention mechanism rather than simple concatenation or averaging.

Key Contributions

1. A framework unifying three components for interest-aware multiple clustering: learnable textual proxies, dynamic candidate refinement, and adaptive feature fusion. The authors state this is the first framework to unify these three elements.
2. A dual-constraint proxy optimization mechanism: a user interest constraint that aligns proxies with the centroid of candidate concept embeddings, and a concept discrimination constraint that uses contrastive learning with hard example mining to sharpen cluster separability.
3. A dynamic candidate management loop: candidate words generated by GPT-4 are scored every R epochs against K-means cluster centroids computed over proxy embeddings; the top-K candidates are retained and re-embedded, shrinking the candidate set from an initial 2^β M entries (β = E/R) to M, matching the number of ground-truth classes.
4. Theoretical and empirical support: Proposition 1 bounds proxy divergence under candidate updates, and Theorem 1 shows how visual features scale the gradient of the alignment loss with respect to the query projection matrix; extensive experiments across all publicly available multiple-clustering tasks show state-of-the-art results.

Main Findings

• Best results across all reported benchmarks: In Table 1, Multi-DProxy achieves the top NMI and RI on every dataset and every clustering object reported, including Fruit Color (NMI 1.0000, RI 1.0000), Fruit Species (NMI 1.0000, RI 1.0000), Card Order (NMI 0.5319, RI 0.9101), Card Suits (NMI 0.5008, RI 0.8848), CMUface Glass (NMI 0.7739, RI 0.8381), CMUface Identity (NMI 0.7609, RI 0.9849), CMUface Pose (NMI 0.6646, RI 0.8991), Stanford Cars Color (NMI 0.7610, RI 0.9403), Flowers Color (NMI 0.7101, RI 0.8939), and CIFAR-10 Type (NMI 0.5863, RI 0.7684).
• Gains over the strongest baselines: Multi-Sub, the next-best method in Table 1, reaches NMI 0.6654 / RI 0.8821 on Fruit360 Color and NMI 0.4828 / RI 0.7096 on CIFAR-10 Environment, versus 0.7058 / 0.8855 and 0.5431 / 0.7204 for Multi-DProxy.
• Beats zero-shot CLIP in nearly all cases: Table 2 compares zero-shot CLIP_GPT (candidate labels from GPT-4) and CLIP_label (ground-truth labels, described as an unfair upper-bound setting). Multi-DProxy outperforms CLIP_GPT in nearly all cases and even exceeds CLIP_label in certain cases, such as clustering by color on Fruit360 (CLIP_label NMI 0.5746 / RI 0.7673 versus Multi-DProxy 0.7058 / 0.8855). CLIP_GPT and CLIP_label perform identically on the Card dataset because GPT-4's candidate labels match the ground truth exactly.
• Every component contributes, fusion contributes most: The ablation variants (w/o-Dynamic, w/o-UConstraints, w/o-CConstraints, w/o-GFusion) all reduce performance when removed. Removing the Gated Cross-Modal Fusion component has the most significant impact.
• Modalities alone are weaker than the fused representation: Table 3 shows text-only (-T) and vision-only (-V) variants underperform the full model, for example on Fruit (0.7639 / 0.7719 for -T, 0.7421 / 0.7471 for -V, versus 1.0000 / 1.0000 for the original), and on CIFAR-10 (0.5030 / 0.6875 and 0.4925 / 0.6789 versus 0.5647 / 0.7444).
• Stronger multi-modal encoders help; the choice of LLM barely matters: Replacing CLIP with ALIGN or BLIP2 improves results, for example Fruit360 Color ALIGN-GPT4 NMI 0.7289 / RI 0.8998 versus CLIP-GPT4 NMI 0.7058 / RI 0.8855. Swapping GPT4 for GPT4o or DeepSeekV3 changes results only marginally, since the LLM is used only to generate candidate words.
• Efficiency advantage: The authors report that Multi-DProxy achieves significantly higher efficiency than the sub-optimal baselines Multi-Sub and Multi-MaP, based on average running time across two datasets (Figure 5). Specific timing numbers are not given in the provided text.
• Training configuration: Models are trained for E = 1000 epochs with the Adam optimizer at momentum 0.9; the candidate update interval R is tuned from {100, 200, 500}; τ_α and σ are tuned from {0.1, 0.2, 0.3, 0.4, 0.5}; experiments run on NVIDIA RTX 4090 GPUs.

Methodology in Plain English

The system starts from CLIP, whose image and text encoders are kept frozen. For each image, a placeholder token "*" is used to create an initial proxy embedding, and GPT-4 generates a wide list of candidate words for the user's stated interest (for example, "color"), with 2^β M candidates initially, where β = E/R.

Each proxy is then built as a weighted combination of the candidate word embeddings, using an attention-style weighting that expresses how much each candidate contributes. This makes the proxy a learnable representation sitting between the fixed vocabulary and the image features.

Visual features and text-plus-proxy features are passed through a stack of layers that attend to each other in both directions. After each attention step, a sigmoid gate decides how much of the attended information to mix in, and a feed-forward block with layer normalization refines the result. At the end, a learned weight λ, computed from the cosine similarity between the final text and visual vectors scaled by a learned temperature τ (initialized at 0.1), blends the two modalities into a single fused feature used for clustering.

Training combines three losses: an alignment loss that pulls the fused feature toward the visual feature of the same sample, a user interest loss that pulls each proxy toward the centroid of the candidate embeddings, and a contrastive loss that penalizes similarity between a sample's fused feature and the proxies of other clusters. The two constraint weights follow schedules over training: α(t) = min(0.5, 0.1 + 0.4 t/E) and β(t) = 0.1 × (1 − cos(π t / E)).

Every R epochs, the candidate words are re-examined: proxies are clustered with K-means into M groups, centroids are computed, each candidate word is scored by its average cosine similarity to all centroids, and the top K = |C|/2 candidates are kept and re-embedded. Over training, the vocabulary narrows from broad generic terms to dataset-specific ones. Clustering is finally performed with K-means on the fused features.

Why This Matters

• Research impact: The paper reframes multiple clustering as an interest-alignment problem rather than an exhaustive partition-generation problem, and shows that making the textual side of a vision-language model learnable and self-refining can outperform both fixed prompt lists and, in some cases, zero-shot classification using ground-truth label names.
• Real-world applications (implied by the framework, not enumerated in the paper):
  • Product catalog organization where a merchandiser wants to group items by color, style, or material on demand.
  • Photo library or media asset management that can regroup a collection by emotion, scene, or person on request.
  • Scientific or biological image collections where a researcher wants to view specimens grouped by species, habitat, or visual trait.
  • Content moderation or dataset curation, where a curator needs different views of the same image pool to find relevant subsets without manually reviewing every generated partition.
• Industry relevance: The framework plugs into existing frozen CLIP-family encoders and off-the-shelf LLMs, and the reported efficiency advantage over Multi-Sub and Multi-MaP suggests it is practical where repeated clustering over large image collections is needed. The finding that stronger multi-modal encoders yield better results gives a clear upgrade path as better encoders become available.

Future Directions

• Extending beyond two modalities: The current design fuses exactly one visual and one textual stream; adding audio, video, or tabular signals is an open extension.
• Reducing dependence on the number of ground-truth classes M: Both the K-means step and the candidate reduction to M candidates rely on M, so learning or estimating M automatically would broaden applicability.
• Deeper evaluation of dynamic candidate management: The paper reports the average ablation effect rather than per-update-cycle behavior; how candidate sets evolve over the R intervals and how sensitive results are to R across more datasets are not fully reported.
• Testing under weaker or noisier LLM candidate generation: The paper finds LLM choice matters little (GPT4, GPT4o, DeepSeekV3 give nearly identical results), which raises the question of how robust the refinement loop is when the initial vocabulary is poor or adversarial.

Target Audience

Researchers and practitioners working on unsupervised representation learning, multi-modal retrieval, and vision-language alignment, particularly those building systems that need to produce several different valid groupings of the same data on demand. It is also relevant to engineers who want to steer a frozen CLIP-style model toward a user-specified concept without fine-tuning the encoders from scratch. Readers without a background in contrastive learning, attention mechanisms, or clustering metrics will find the theoretical sections and loss formulations demanding.

Authors’ abstract

Multiple clustering aims to discover diverse latent structures from different perspectives, yet existing methods generate exhaustive clusterings without discerning user interest, necessitating laborious manual screening. Current multi-modal solutions suffer from static semantic rigidity: predefined candidate words fail to adapt to dataset-specific concepts, and fixed fusion strategies ignore evolving feature interactions. To overcome these limitations, we propose Multi-DProxy, a novel multi-modal dynamic proxy learning framework that leverages cross-modal alignment through learnable textual proxies. Multi-DProxy introduces 1) gated cross-modal fusion that synthesizes discriminative joint representations by adaptively modeling feature interactions. 2) dual-constraint proxy optimization where user interest constraints enforce semantic consistency with domain concepts while concept constraints employ hard example mining to enhance cluster discrimination. 3) dynamic candidate management that refines textual proxies through iterative clustering feedback. Therefore, Multi-DProxy not only effectively captures a user's interest through proxies but also enables the identification of relevant clusterings with greater precision. Extensive experiments demonstrate state-of-the-art performance with significant improvements over existing methods across a broad set of multi-clustering benchmarks.