Free-Grained Hierarchical Visual Recognition

AI summary

Overview

• Research area: Computer vision, specifically hierarchical image recognition and mixed-granularity (semi-/weakly-supervised) learning.
• Technical level: Intermediate. Readers should know basic classification and contrastive-learning ideas; the hierarchy and granularity framing is explained from scratch.
• Scope: The paper defines "free-grained" hierarchical recognition, builds benchmarks for it, proposes two training methods for incomplete labels, and compares two strategies for deciding how deep a model should predict.

What This Paper Is About

Hierarchical recognition normally assumes every training image is annotated along its full taxonomy path, from a broad category down to a specific one, but real annotations are messier: a distant bird may be labeled only "bird" while a clear close-up may justify "bald eagle." The authors introduce free-grained training, where labels may appear at any level of a taxonomy and may differ across instances, and free-grained inference, where the model chooses how deep to predict so that it returns a reliable coarse label when a fine-grained one is uncertain. The goal is to learn taxonomy-consistent predictions from incomplete, uneven supervision.

Key Contributions

1. A new task definition: free-grained hierarchical visual recognition, covering both training with mixed-granularity labels and inference that adaptively selects prediction depth.
2. New benchmarks: foundation-based variants (ImageNet-F, iNat21-mini-F, CUB-F) that mimic realistic annotation difficulty via label pruning, plus randomized variants (CUB-Rand, Aircraft-Rand) for controlled label availability, along with a re-curated three-level ImageNet hierarchy called ImageNet-3L.
3. Two training methods: text-guided pseudo-attributes, which add image-description supervision to compensate for missing fine labels, and taxonomy-guided semi-supervised learning (Taxon-SSL), which treats missing level labels as unlabeled data and enforces cross-level consistency.
4. An inference study: a comparison showing that consistency-based stopping yields more reliable and deeper correct predictions than confidence-based stopping.

Main Findings

• Existing methods collapse under mixed granularity: When trained with free-grained labels, prior hierarchical classifiers lose up to 40% in full-path accuracy on iNat21-mini. The prior state-of-the-art H-CAST drops from 84.9% to 45.1% on CUB-F and from 64.9% to 25.6% on iNat21-mini-F (Fig. 5a); the paper summarizes this as a 19–40 percentage-point drop across datasets.
• Text-Attr works best at scale on ImageNet-F: On ImageNet-F, Text-Attr (H-CAST) reaches 63.20% full-path accuracy versus 57.59% for H-CAST and 37.79% for HRN, with the lowest TICE of 18.58. Text-Attr on H-ViT alone reaches 55.48% FPA without H-CAST's visual grouping modules.
• Taxon-SSL works best in the fine-grained biological domain: On iNat21-mini-F, Taxon-SSL reaches 31.74% FPA, ahead of H-CAST (25.63%) and HRN (17.03%), and also cuts TICE to 37.31 from H-CAST's 47.17. Text-Attr methods land at 27.88–29.74% FPA there, which the authors attribute to restricted textual diversity in that domain.
• Combining the two helps modestly: Taxon-SSL + Text-Attr reaches 49.65% FPA on ImageNet-F and 31.93% on iNat21-mini-F, giving consistent but modest gains over either method alone.
• The methods are complementary by class size: A per-class analysis on ImageNet-F shows Text-Attr tends to win on classes with fewer available training samples, while Taxon-SSL tends to win on classes with more training samples.
• Text guidance improves where the model looks: Saliency comparisons show Taxon-SSL drifting to semantically irrelevant regions under sparse supervision (for example assigning "clothing" at the basic level and "saxophone" at the fine level), while Text-Attr attends to the relevant object and keeps predictions semantically plausible.
• Consistency-based inference beats confidence-based inference: Confidence-based stopping, using a threshold of 0.9 chosen from [0.85, 0.99], often stops prematurely because probability mass is split among similar sibling classes; consistency-based stopping requires no threshold tuning and reaches deeper correct levels. Under that rule, Text-Attr (H-CAST) produces the most taxonomy-consistent outputs.
• Not reported in the provided content: numeric results for CUB-F and for the randomized variants (CUB-Rand, Aircraft-Rand) appear only as references to appendix sections (D.1, D.2), not in the main text shown here.

Methodology in Plain English

The authors first fix the taxonomy problem. ImageNet's WordNet hierarchy is inconsistent (classes span 5–19 levels, and 30% of classes have multiple valid paths), so they rebuild it into a clean three-level tree of 20 basic, 127 subordinate, and 505 fine-grained classes (ImageNet-3L), following cognitive-psychology ideas about a natural "basic" level.

To simulate realistic incomplete annotation, they prune labels using foundation models: CLIP for ImageNet-F, BioCLIP for iNat21-mini-F and CUB-F. The basic label is always kept; the subordinate label is kept only if the model's subordinate prediction is correct; the fine-grained label is kept only if both the subordinate and fine predictions are correct. They also remove some subordinate labels based on per-class fine-grained removal rates. Only labels are removed, so no new semantic information is added and difficulty may even increase.

For controlled experiments, they randomly prune labels from CUB and Aircraft, denoting availability as a-b-c percentages for basic, subordinate, and fine labels.

To adapt prior hierarchical classifiers to this setting, they apply the hierarchical loss only at levels where a label exists. Their first method, Text-Attr, generates a visual description for each image with a frozen vision-language model (Llama-3.2-11B), encodes it with CLIP's text encoder (capped at 100 generated tokens, truncated to CLIP's 77-token limit), and aligns the image embedding with the text embedding using a contrastive loss, so the encoder picks up shared visual attributes across levels.

Their second method, Taxon-SSL, generalizes CHMatch beyond its two-level form: it generates pseudo-labels at every taxonomy level and builds a taxonomy-aligned affinity where two samples count as positives only if they agree at all levels, then uses a contrastive objective to pull consistent pairs together and push inconsistent ones apart.

Evaluation uses top-1 accuracy per level, a Tree-based InConsistency Error rate (TICE), and Full-Path Accuracy (FPA), the primary metric. Comparisons are against HRN (ResNet-50 backbone) and H-CAST, with H-ViT (ViT-Small-based) as the backbone for the proposed methods; models train for 100 epochs except ImageNet-F, which trains for 200.

Why This Matters

Free-grained recognition reframes hierarchical classification to match how labels actually arise: image difficulty, annotator expertise, cost, and shifting labeling protocols all produce labels at varying depth. Instead of assuming tidy full-path annotations that are expensive to collect, the proposed setting lets practitioners use the labels they already have.

Real-world applications:

• Biodiversity monitoring: Citizen-science and camera-trap images often carry only coarse labels, as in iNat21-mini-style species data.
• Expert-to-layperson content delivery: A model can return "bald eagle" to an expert or "bird" to a general user from the same system.
• Product and retail catalogs: Merchandise taxonomies are deep and inconsistently annotated; a model that predicts at the safe depth avoids confidently wrong fine labels.
• Medical or scientific imaging: Where only specialists can provide fine labels, coarse labels from non-experts can still be used for training.

Industry relevance: the results give practical guidance on when to use text-generated auxiliary supervision (large, visually diverse datasets like ImageNet-F) versus structural semi-supervised propagation (domains with many visually similar classes like iNat21-mini-F), and show that a correct coarse prediction is often preferable to an incorrect fine one.

Future Directions

• Class- and level-wise imbalance: The authors explicitly leave these unaddressed and identify them as future work.
• Better label pruning: They suggest exploring pruning beyond CLIP, for example pruning based on models or ensembles that mispredict consistently, to better approximate intrinsic annotation difficulty.
• Scaling the benchmarks: CUB and Aircraft are small-scale and iNat21-mini is confined to a single biological domain, so broader large-scale free-grain benchmarks remain open.
• Stronger robustness under missing supervision: Because existing methods still degrade sharply and the improvements over state of the art are described as strong baselines rather than solved, more robust free-grained learning is an open problem; the appendix sections on the unrefined WordNet hierarchy and further ablations point in this direction.

Target Audience

Researchers and graduate students in computer vision working on hierarchical classification, fine-grained recognition, semi-supervised and weakly supervised learning, or long-tailed recognition. It is also relevant to practitioners who must train models on taxonomies with incomplete, unevenly annotated data, and to those interested in label-efficient training pipelines that use vision-language models as auxiliary supervision.

Authors’ abstract

Hierarchical image recognition seeks to predict class labels along a semantic taxonomy, from broad categories to specific ones, typically under the tidy assumption that every training image is fully annotated along its taxonomy path. Reality is messier: A distant bird may be labeled only bird, while a clear close-up may justify bald eagle. We introduce free-grain training, where labels may appear at any level of the taxonomy and models must learn consistent hierarchical predictions from incomplete, mixed-granularity supervision. We build benchmark datasets with varying label granularity and show that existing hierarchical methods deteriorate sharply in this setting. To make up for missing supervision, we propose two simple solutions: One adds broad text-based supervision that captures visual attributes, and the other treats missing labels at specific taxonomy levels as a semi-supervised learning problem. We also study free-grained inference, where the model chooses how deep to predict, returning a reliable coarse label when a fine-grained one is uncertain. Together, our task, datasets, and methods move hierarchical recognition closer to the way labels arise in the real world.