Multi-Modal Representation Learning via Semi-Supervised Rate Reduction for Generalized Category Discovery

AI summary

Overview

Research area: Computer vision, specifically Generalized Category Discovery (GCD) and multi-modal representation learning with vision-language models.

Technical level: Advanced. The paper builds on maximal coding rate reduction theory, contrastive learning losses, and CLIP-based dual-branch training pipelines, and assumes familiarity with those frameworks.

Scope: The paper proposes SSR²-GCD, a Semi-Supervised Rate Reduction framework for Generalized Category Discovery that learns structured image and text representations through intra-modality alignment rather than relying mainly on inter-modality alignment.

What This Paper Is About

Generalized Category Discovery asks a model to label unlabeled images that may belong either to categories it saw during training (known) or to entirely new categories it has never seen (unknown), using only partial labels for the known categories. Existing multi-modal GCD methods concentrate on aligning image and text modalities to each other (inter-modality alignment) but pay little attention to the structure within each modality (intra-modality alignment). This paper argues that this imbalance causes "imbalanced compression" of the embedding space, where known categories get squeezed together far more than unknown ones, and proposes a loss based on the maximal coding rate reduction principle to fix it.

Key Contributions

1. SSR²-GCD framework: A Semi-Supervised Rate Reduction framework for GCD that learns representations with a desired underlying distribution structure by explicitly addressing intra-modality alignment. The authors state this is the first work to address the imbalanced compression issue in contrastive-based representation learning for GCD.

2. The SSR² loss: An objective combining an expansion term R(Z) with two compression terms R_c^s(Z, Y*) for labeled data and R_c^u(Z, Y) for unlabeled data, applied separately to image embeddings and text embeddings. This is claimed to compress known and unknown categories in a balanced manner.

3. Retrieval-based Text Aggregation (RTA): A strategy that aggregates the top-c most similar tag and attribute prompts (with c = 4 and α = 0.5) instead of using a single concatenated long prompt, motivated by CLIP's limitation in handling prompts exceeding 20 tokens.

4. Extensive empirical validation: Experiments on eight benchmark datasets (four generic, four fine-grained) plus ablations on representation learning, intra-modal consistency, and embedding rank, reported as averages over 3 trials.

Main Findings

• Best multi-modal results across all tested datasets: SSR²-GCD achieved "All" accuracy of 92.1% on ImageNet-100, 66.7% on ImageNet-1k, 98.5% on CIFAR-10, 86.4% on CIFAR-100, 78.3% on CUB, 89.2% on Stanford Cars, 95.7% on Oxford Pets, and 93.5% on Flowers102. The authors report it consistently outperforms all other multi-modal counterparts.

• Largest margins on fine-grained datasets: On Stanford Cars and Flowers102, SSR²-GCD outperformed all other baselines by 3.1% and 6.3% respectively on "All" categories, reaching 89.2% and 93.5%.

• Narrowed Old-vs-New gap: The authors report that SSR²-GCD reduces the accuracy gap between known ("Old") and unknown ("New") categories, for example 96.0% vs. 90.2% on ImageNet-100 and 93.3% vs. 93.9% on Flowers102.

• Intra-modal losses beat inter-modal alignment: In the representation-learning comparison (Table 3, "All" accuracy), using inter-modal loss L_CLIP alone gave 98.3% / 86.0% / 76.7% / 87.0% / 94.1% / 89.7% on CIFAR-10, CIFAR-100, CUB, Stanford Cars, Oxford Pets, and Flowers102, versus 98.5% / 86.4% / 78.3% / 89.2% / 95.7% / 93.5% with the proposed L_SSR². Frozen CLIP gave 97.9% / 84.1% / 74.5% / 86.0% / 91.9% / 87.4%, and intra-modal L_con gave 98.4% / 86.7% / 77.5% / 87.9% / 94.9% / 91.8%.

• Combining inter- and intra-modal losses can hurt: L_CLIP + L_con scored 98.2% / 86.3% / 78.0% / 86.7% / 95.0% / 90.9% across the six datasets, and L_CLIP + L_SSR² scored 98.3% / 86.1% / 77.2% / 88.1% / 95.0% / 92.9%. The authors report L_SSR² surpassed L_CLIP + L_SSR² on all datasets, suggesting explicit inter-modal alignment can deteriorate intra-modal alignment.

• Inter-modal alignment alone causes intra-modal misalignment: Distribution plots of image-text, image-image, and text-text pairwise similarities on Flowers102 across epochs 0, 10, and 200 show that L_SSR² aligns intra-modal similarity distributions during warm-up and nearly closes all gaps by the end of training, while L_CLIP aligns image-text similarities but leaves image-image and text-text similarities poorly aligned.

• Higher intra-modal consistency: Using the consistency measure ρ (ratio of within-category edge weights to between-category edge weights), L_SSR² achieved higher ρ than L_CLIP on Flowers102 and Stanford Cars. The curve for L_CLIP + L_SSR² rose in parallel with L_SSR² early in training but eventually converged to the L_CLIP curve.

• Balanced compression confirmed by rank analysis: With L_con, the average numerical rank of image embeddings for "Old" categories dropped dramatically and fell well below that of "New" categories; with L_SSR², the average ranks of both "Old" and "New" categories were preserved well, avoiding collapse.

• Behavior on imbalanced data: Training on the entire class-imbalanced Flowers102 dataset produced larger ranks for larger categories, consistent with the scaled compression term learning balanced embeddings where bigger categories span more dimensions with smaller singular values.

• Ablation results (partial): On Stanford Cars, the baseline (single classifier, most similar tag as pseudo-text, frozen CLIP) reached 75.2% / 85.4% / 71.8% (All/Old/New) and adding dual-branch classifiers raised this to 81.7% / 90.3% / 77.1%. On Flowers102, the baseline reached 78.3% / 88.1% / 72.5% and the dual-branch version reached 83.9% / 88.3% / 81.3%. The remaining ablation rows are not included in the provided content.

• Comparison to zero-shot CLIP: The paper notes CLIP performs poorly out of domain on Flowers102, achieving 70.4% in zero-shot classification, while SSR²-GCD refines CLIP representations to reach 93.5% on "All" categories.

• HypCD exception on CUB: HypCD (with SelEx backbone) achieved the highest accuracy on CUB, which the authors attribute to its hyperbolic embedding space being complementary to SSR²-GCD rather than competing with it.

Methodology in Plain English

The approach keeps the usual multi-modal GCD pipeline (text generation, representation learning, clustering) but changes how representations are learned.

Text side. Tag and attribute lexicons are built and CLIP-H/14 is used to retrieve the most similar candidates for each image. Instead of concatenating many prompts into one long string, the system embeds the top-c most similar tags and attributes separately and combines them with a weighted average: the single most similar candidate receives weight 1 − α, and the remaining candidates share weight α/(c − 1). The paper uses α = 0.5 and c = 4.

Representation side. The core loss has three parts. One term expands the overall embedding space of the mini-batch; two other terms compress embeddings belonging to the same category into low-dimensional subspaces — one using ground-truth labels for known categories, one using pseudo-labels predicted by the classifiers for unlabeled data. This structure is meant to compress known and unknown categories in balanced fashion, in contrast to contrastive losses that pull labeled (known) data together much more aggressively. The same loss is applied separately to image embeddings and text embeddings (using CLIP-B/16 encoders).

Training procedure. There are two stages. During warm-up, the unlabeled compression term is removed to avoid being misled by untrained-classifier pseudo-labels, and the standard SimGCD-style classification loss (cross-entropy on labeled data, self-distillation, and an entropy regularizer) trains separate image and text classifiers. In the alignment stage, the unlabeled compression term returns and a co-teaching loss from TextGCD aligns the pseudo-label orderings of the two classifiers. At inference, final predictions come from the argmax of the image and text classifier outputs summed together.

Evaluation. ACC is computed with Hungarian matching and reported for "All," "Old," and "New" categories, averaged over 3 trials. Following the GCD protocol, half the samples of known categories form the labeled set.

Why This Matters

The paper challenges the assumption that more inter-modality alignment is always better in multi-modal GCD. It shows that explicitly forcing images and texts together can damage the within-modality structure that clustering algorithms depend on, and it offers a principled alternative rooted in coding rate reduction theory.

Real-world applications — the paper itself does not enumerate specific applications; benchmarks and framing suggest the following as plausible settings:

• Open-world visual exploration where data is a mix of known and novel categories, such as cataloging new species or new product types.
• Fine-grained recognition tasks (cars, birds, pets, flowers), where discovering visually similar new subcategories is the central difficulty and where the method shows its largest gains.
• Domain-shifted deployment, where a model trained on one visual domain must adapt to a new one, as illustrated by CLIP's weak 70.4% zero-shot result on Flowers102 versus 93.5% after refinement.
• Any labeling pipeline that starts with a partially labeled dataset and needs to surface remaining categories without knowing their names in advance.

Industry relevance: The method reuses frozen or lightly fine-tuned CLIP backbones and standard SimGCD-style classifier heads, and its main added component is a new loss, so it can be dropped into existing multi-modal GCD pipelines. The authors provide code at a public GitHub repository. Companies working on search, cataloging, or content moderation with partially labeled data could apply the framework to reduce labeling cost.

Future Directions

• Combining with alternative embedding geometries. The authors explicitly note that HypCD's hyperbolic embedding space is complementary to SSR²-GCD, since HypCD achieved the highest CUB accuracy, raising the question of whether rate reduction and hyperbolic representation learning can be combined.

• Estimating the number of unknown categories. The problem setup assumes the total number of categories K is given or can be estimated; the paper does not propose a method for estimating it.

• Extending beyond two modalities. The current framework handles images and text; whether SSR² generalizes to additional modalities is not explored.

• Improving prompt construction further. The RTA strategy was motivated by CLIP's failure on prompts exceeding 20 tokens. Whether a different text encoder or a long-prompt-capable model would remove this constraint and change the design is untested within this paper.

• Understanding why inter-modal losses degrade intra-modal structure. The paper demonstrates the phenomenon via similarity distributions and the ρ measure but frames the underlying mechanism as an observation rather than a fully resolved theoretical account.

Target Audience

Researchers and graduate students working on open-set recognition, novel class discovery, semi-supervised learning, and vision-language representation learning. It is also relevant to practitioners building clustering or labeling pipelines on partially annotated image collections who are comfortable with contrastive learning and CLIP-based architectures. Readers without background in information theory (specifically coding rate reduction) and GCD benchmarks will need to consult the cited prior work to follow the loss derivations.

Authors’ abstract

Generalized Category Discovery (GCD) aims to identify both known and unknown categories, with only partial labels given for the known categories, posing a challenging open-set recognition problem. State-of-the-art approaches for GCD task are usually built on multi-modality representation learning, which is heavily dependent upon inter-modality alignment. However, few of them cast a proper intra-modality alignment to generate a desired underlying structure of representation distributions. In this paper, we propose a novel and effective multi-modal representation learning framework for GCD via Semi-Supervised Rate Reduction, called SSR$^2$-GCD, to learn cross-modality representations with desired structural properties based on emphasizing to properly align intra-modality relationships. Moreover, to boost knowledge transfer, we integrate prompt candidates by leveraging the inter-modal alignment offered by Vision Language Models. We conduct extensive experiments on generic and fine-grained benchmark datasets demonstrating superior performance of our approach.