SEAL: Semantic-Aware Hierarchical Learning for Generalized Category Discovery

AI summary

Overview

Research area: Computer vision, specifically open-world / semi-supervised visual recognition — Generalized Category Discovery (GCD), fine-grained classification, contrastive representation learning, and hierarchical (taxonomic) label learning.

Technical level: Intermediate to Advanced. The paper assumes familiarity with contrastive learning (InfoNCE), mutual information, Vision Transformer backbones, parametric prototype classifiers, and clustering-accuracy evaluation. The information-theoretic motivation and the multi-level training design are not beginner material.

One-sentence scope: The paper proposes SEAL, a single-stage framework that replaces abstract, manually designed hierarchies in GCD with naturally occurring semantic hierarchies (coarse-to-fine labels) and shows state-of-the-art accuracy on fine-grained benchmarks such as the SSB benchmark, Oxford-Pet, and Herbarium19.

Authors: Zhenqi He, Yuanpei Liu, Kai Han (Visual AI Lab, The University of Hong Kong). Posted as arXiv:2510.18740v1 [cs.CV] on 21 Oct 2025. Code/project page: https://visual-ai.github.io/seal/

What This Paper Is About

Generalized Category Discovery starts from a partially labelled dataset and asks a model to categorize every unlabelled image — both the classes it has already seen and classes it has never been trained on. Prior GCD methods either use only a single level of semantics, or they invent "abstract" hierarchies (implicit binary trees, iteratively merged partitions, hand-defined upper/lower levels, or hyperbolic embeddings) that can introduce noise and errors, for example wrongly merging "Red Fox" into one category or splitting "Siberian Tiger" and "Bengal" apart. SEAL's goal is to use the semantic taxonomies that already exist in the real world for free, instead of manufacturing hierarchies from data or hyperparameters.

Key Contributions

1. A semantic-aware hierarchical framework for GCD (SEAL). The authors state this is the first exploration of using inherent semantic hierarchies for GCD, and it is trained end-to-end in a single stage rather than as a multi-stage pipeline.
2. Hierarchical Semantic-Guided Soft Contrastive Learning. A contrastive objective that uses hierarchical similarity to produce informative soft negatives, addressing the limitation of conventional contrastive losses that treat all negatives equally.
3. A Cross-Granularity Consistency (CGC) module. A self-distillation term that aligns class predictions across different levels of granularity so that, for example, a model does not simultaneously predict "Shiba" at one level and "Cat" at another for the same image. It uses a dynamically updated transition matrix M_h (initialized one-hot for known fine-grained classes and uniform for novel classes, then refined during training).
4. A theoretical motivation and broad empirical validation. Using the chain rule of mutual information, the authors show that incorporating semantic hierarchical labels yields a strictly tighter bound on mutual information, and the method is then tested on five reported benchmarks plus generic datasets in the appendix.

Main Findings

• State-of-the-art on the SSB benchmark with DINOv2 features: SEAL reaches an average "All" accuracy of 76.3%, outperforming the previous best method, DebGCD, by 1.4%. SEAL's per-dataset DINOv2 "All/Old/New" scores are 76.7/78.3/75.9 on CUB, 77.7/88.7/72.4 on Stanford Cars, and 74.6/73.2/75.3 on FGVC-Aircraft.
• New state of the art on Herbarium19: SEAL reaches 46.9% on "All" accuracy, surpassing the previous best method, μGCD, by 1.1% (45.8% Old, 48.2% New, versus μGCD's 61.9% Old and 37.2% New).
• Competitive on Oxford-Pet: SEAL scores 92.9% All / 88.9% Old / 95.0% New, above the SimGCD and InfoSieve baselines but slightly below DebGCD's 93.0% All.
• CUB is the acknowledged weak spot: under DINOv1, SEAL obtains 66.2% All on CUB, slightly behind DebGCD (66.3%) and the non-parametric InfoSieve (69.4%). The authors attribute this gap to the human-annotated nature of bird taxonomy, which may contain inconsistencies absent from more systematically defined hierarchies such as vehicles and aircraft.
• Strong performance on DINOv1 SSB as well: SEAL is highest on both Stanford Cars (65.3/79.3/58.5) and FGVC-Aircraft (62.0/65.3/60.4), with an average "All" of 64.5% versus DebGCD's 64.4%.
• Every component contributes, and they compound: starting from the baseline (53.8 All on Stanford Cars, 60.3 on CUB, 54.2 on FGVC-Aircraft), adding hierarchical learning, then consistency self-distillation, then semantic-guided HSCL gives final gains of +11.5% on Stanford Cars, +5.9% on CUB and +7.8% on FGVC-Aircraft in the ablation table (the body text states 5.7% for CUB). The largest single-level gains are on "New" classes: +13.5% on Stanford Cars, +8.6% on FGVC-Aircraft, +5.5% on CUB.
• Two hyperparameters dominate the tuning: on Stanford Cars, best performance is achieved with consistency temperature τ_c = 0.75 and soft-negative controller λ_s = 1.0, giving 65.3/79.3/58.5 on the unlabelled set and 66.4/77.3/55.9 on the held-out validation split.
• The framework works across alternative semantic dimensions: with a DINOv2 backbone on Stanford Cars, a vehicle-type hierarchy (SUV/Van/Coupe) gives 77.7/88.7/72.4 and a vehicle-brand hierarchy (Audi/BMW) gives 77.1/89.0/71.3, both above SimGCD, μGCD and DebGCD. The paper notes the labels can come from curated taxonomies or be generated by LLMs.
• Qualitative evidence of hierarchy-aware features: a t-SNE visualization over 20 randomly sampled Stanford Cars categories (10 Old, 10 New) shows tighter, better-separated clusters than the baseline, with visually diverse members of the broader "Cab" group remaining

Authors’ abstract

This paper investigates the problem of Generalized Category Discovery (GCD). Given a partially labelled dataset, GCD aims to categorize all unlabelled images, regardless of whether they belong to known or unknown classes. Existing approaches typically depend on either single-level semantics or manually designed abstract hierarchies, which limit their generalizability and scalability. To address these limitations, we introduce a SEmantic-aware hierArchical Learning framework (SEAL), guided by naturally occurring and easily accessible hierarchical structures. Within SEAL, we propose a Hierarchical Semantic-Guided Soft Contrastive Learning approach that exploits hierarchical similarity to generate informative soft negatives, addressing the limitations of conventional contrastive losses that treat all negatives equally. Furthermore, a Cross-Granularity Consistency (CGC) module is designed to align the predictions from different levels of granularity. SEAL consistently achieves state-of-the-art performance on fine-grained benchmarks, including the SSB benchmark, Oxford-Pet, and the Herbarium19 dataset, and further demonstrates generalization on coarse-grained datasets. Project page: https://visual-ai.github.io/seal/