Generalized Category Discovery under Domain Shift: A Frequency Domain Perspective

AI summary

Overview

• Research area: Computer vision, specifically Generalized Category Discovery (GCD) and unsupervised domain adaptation.
• Technical level: Advanced. The paper assumes familiarity with self-supervised contrastive learning, prototype classifiers, Fourier analysis of images, and Gaussian mixture models.
• Scope: The paper proposes and evaluates FREE, a frequency-domain framework for discovering both known and novel categories when unlabeled data also comes from unseen domains (the DS_GCD setting), tested on DomainNet and the Corrupted Semantic Shift Benchmark (SSB-C).

What This Paper Is About

Generalized Category Discovery (GCD) asks a model to use labeled examples of known categories to cluster unlabeled images that may belong to either known or brand-new categories. Existing GCD methods assume that labeled and unlabeled data come from the same domain, and the paper shows their accuracy collapses when the unlabeled images also come from a different, unseen domain. The authors define this harder setting as Domain-Shifted Generalized Category Discovery (DS_GCD) and build a method that uses frequency-domain cues, particularly the amplitude component of images, to separate domains and adapt across them.

Key Contributions

1. A new problem setting and a frequency-domain solution. The authors formalize DS_GCD, where unlabeled data contains both unknown categories and samples from unknown domains, and propose the Frequency-guided Generalized Category Discovery framework (FREE) to address it.
2. Frequency-based domain separation and perturbation. FREE introduces a Frequency-Based Domain Separation (FDS) strategy that partitions samples into known and unknown domains by measuring amplitude differences, plus a Cross-Domain Frequency Perturbation (CDFP) strategy and an Intra-Domain Frequency Perturbation (IDFP) strategy for adapting to new distributions.
3. Extended contrastive and clustering objectives. The self-supervised contrastive objective and semantic clustering loss from the SimGCD baseline are extended to incorporate frequency-perturbed samples as additional positives and to use sharpened predictions from frequency-augmented views as clustering targets.
4. Clustering Difficulty-Aware Sampling (CDAS). A resampling mechanism scores each class by intra-class compactness and inter-class separability and samples harder-to-cluster categories more often, plus extensive experiments showing gains over state-of-the-art methods on all tested datasets.

Main Findings

• Domain shift severely damages existing GCD methods. The authors report that existing GCD approaches suffer significant performance degradation under domain shift, and that unlabeled samples from unknown domains can even impair clustering accuracy within the known domain.
• FREE outperforms the strongest baseline HiLo on SSB-C corrupted domains. On the corrupted (unknown) domains of CUB-C, Scars-C, and FGVC-C, FREE improves clustering accuracy on all categories by 3.7%, 3.3%, and 3.8% respectively over HiLo. On FGVC-C corrupted data, FREE reaches 35.0 (All), 32.4 (Old), and 38.9 (New) versus HiLo's 31.2, 29.0, and 33.4.
• Gains also appear in known (clean) domains. FREE improves clustering within the clean domains as well; on CUB-C original data it reports 60.4 (All), 58.5 (Old), 63.2 (New) against HiLo's 56.8, 54.0, 60.3.
• FREE leads on DomainNet. Using Real as the known domain and Painting as the unknown domain, FREE outperforms HiLo by 3.5% on all categories in the Painting domain (45.6 versus 42.1) and by 3.3% in the Real domain (67.7 versus 64.4).
• Random frequency swapping can hurt. A comparison of FREE, SimGCD+FDA, and SimGCD (Figure 2) shows FREE wins on both final clustering accuracy and convergence speed, while random transformation in SimGCD+FDA can hinder learning of unknown domains and produce negative transfer.
• Component ablation supports each module. With SimGCD as baseline on Real+Painting, the baseline scores 61.3 (Real All) and 34.5 (Painting All). Adding FDS and IDFP raises these to 62.0 and 36.6; adding CDFP instead of IDFP gives 65.6 and 41.6; combining FDS, IDFP, and CDFP gives 66.1 and 42.8; the full FREE with CDAS gives 67.7 and 45.6.
• Class-aware perturbation beats random and class-agnostic variants. Under the same Real+Painting setup, random perturbations yield 62.7 (Real All) and 33.2 (Painting All), a class-agnostic variant yields 65.6 and 43.5, and FREE yields 67.7 and 45.6.
• Domain separation is empirically valid. Density histograms (Figure 4) show most known-domain samples have higher density values and fall on the right of the histogram, while unknown-domain samples tend to have lower density values and fall on the left.
• The paper content is truncated. The provided text ends mid-sentence in the ablation discussion; no conclusion section, limitations discussion, or explicit future work is reported in the available content.

Methodology in Plain English

The researchers start from SimGCD, a GCD method that combines a contrastive loss with a clustering loss over a prototype classifier and an entropy regularizer, and keep that optimization structure while changing how data is prepared and weighted.

1. Splitting the data by style, not labels. Each image is converted with the Fast Fourier Transform, producing an amplitude component (which prior work associates with style) and a phase component (associated with content). For every unlabeled image, the method computes a density score against an anchor set of amplitude representations from labeled known-domain samples, using the K nearest neighbors (K set to 3). A two-component Gaussian mixture model, fitted with Expectation-Maximization, turns this score into a soft probability that a sample is from the known or the unknown domain. This avoids needing domain labels.

2. Mixing styles between domains, class by class. For a known-domain sample predicted as class k, the method finds an unknown-domain sample also predicted as class k with confidence above a threshold (η = 0.9) and swaps the low-frequency amplitude components, then converts back with the inverse FFT. A FIFO memory bank holding the most recent 1024 unknown-domain samples keeps class coverage when a batch lacks a needed class; early in training, unknown-domain samples are chosen randomly instead.

3. Making the model robust within the unknown domain. For an unknown-domain sample, two different augmentations are created, their low-frequency amplitudes are exchanged, and the resulting view is treated as an extra positive pair in contrastive learning and used to produce sharpened pseudo-labels for the clustering loss.

4. Focusing on the hard clusters. Each class gets a difficulty score combining how spread out its features are around its prototype (intra-class compactness) and how similar its prototype is to other prototypes (inter-class separability). A softmax over these scores defines a sampling probability, so classes that are harder to cluster are drawn more often.

The total objective sums the known-domain and unknown-domain losses plus the entropy regularizer. Training uses a DINO-pretrained ViT-B/16 backbone with only the last layer fine-tuned, SGD with a batch size of 256, an initial learning rate of 0.1 under cosine annealing, 200 epochs, β = 0.35 and ε = 0.1, three random seeds, Faiss for nearest-neighbor search, and eight NVIDIA RTX 4090 GPUs.

Why This Matters

• Research impact: The paper argues that standard GCD benchmarks overstate real-world performance because they ignore domain shift, and it provides a benchmark protocol (DomainNet and SSB-C) plus a method that existing GCD, NCD, and UDA approaches are compared against. It also positions frequency-domain adaptation as a controllable alternative to random amplitude swapping.
• Real-world applications:
  • Clinical imaging: the paper's own motivating example, where images from different sources differ in appearance and statistical properties while new disease categories appear unexpectedly.
  • Any deployment collecting images across cameras, sensors, or sites where new object categories keep emerging and domain labels are unavailable.
  • Fine-grained visual cataloguing (birds, cars, aircraft, as in the SSB-C subsets) under degraded capture conditions such as noise, blur, fog, and spatter.
  • Large mixed-source image collections, since DomainNet-scale data (about 600,000 images, 345 categories, six domains) reflects heterogeneous web-sourced imagery.
• Industry relevance: The method needs no domain labels for unlabeled data and only fine-tunes the last layer of a pretrained ViT-B/16, which lowers labeling cost and compute overhead for production systems that must absorb new categories from shifting data streams.

Future Directions

• Extending the truncated evaluation. The provided content cuts off mid-sentence in the ablation analysis, so further reported analysis, conclusions, and any stated limitations are not available and would need the full paper.
• Improving pseudo-label reliability in the unknown domain. FREE depends on confidence thresholds and a memory bank to control noisy pseudo-labels for cross-domain amplitude exchange; how sensitive results are to those choices is not reported in the available content.
• Handling multiple unknown domains explicitly. The paper notes that unknown domains may consist of multiple domains and that all remaining DomainNet domains can be combined into one unified unknown domain, but the provided content does not report a dedicated analysis of many-domain mixtures.
• Reducing dependence on frequency-domain assumptions. Whether amplitude-versus-phase separation holds for modalities and data types outside natural and fine-grained images is not addressed in the available content.

Target Audience

Researchers and graduate students working on category discovery, novel class discovery, open-world recognition, and unsupervised domain adaptation, particularly those comfortable with self-supervised contrastive learning and Fourier-based image manipulation. It is also relevant to applied machine learning engineers who need recognition systems to keep discovering new categories when data sources drift, and to practitioners in medical imaging or other multi-source settings where new categories and new domains appear together. Readers seeking a beginner-level introduction to GCD would need background reading first.

Authors’ abstract

Generalized Category Discovery (GCD) aims to leverage labeled samples from known categories to cluster unlabeled data that may include both known and unknown categories. While existing methods have achieved impressive results under standard conditions, their performance often deteriorates in the presence of distribution shifts. In this paper, we explore a more realistic task: Domain-Shifted Generalized Category Discovery (DS\_GCD), where the unlabeled data includes not only unknown categories but also samples from unknown domains. To tackle this challenge, we propose a \textbf{\underline{F}}requency-guided Gene\textbf{\underline{r}}alized Cat\textbf{\underline{e}}gory Discov\textbf{\underline{e}}ry framework (FREE) that enhances the model's ability to discover categories under distributional shift by leveraging frequency-domain information. Specifically, we first propose a frequency-based domain separation strategy that partitions samples into known and unknown domains by measuring their amplitude differences. We then propose two types of frequency-domain perturbation strategies: a cross-domain strategy, which adapts to new distributions by exchanging amplitude components across domains, and an intra-domain strategy, which enhances robustness to intra-domain variations within the unknown domain. Furthermore, we extend the self-supervised contrastive objective and semantic clustering loss to better guide the training process. Finally, we introduce a clustering-difficulty-aware resampling technique to adaptively focus on harder-to-cluster categories, further enhancing model performance. Extensive experiments demonstrate that our method effectively mitigates the impact of distributional shifts across various benchmark datasets and achieves superior performance in discovering both known and unknown categories.