Multilabel, Fine-Grained, and Hierarchical Recognition

All four produce labels. Their supervision and their errors are not interchangeable.

Multilabel means several labels can be true of one image. The head is per-label or structured, the risk is co-occurrence shortcuts, and evaluation runs per label and per example. Fine-grained means classes differ by subtle local evidence. Resolution must preserve detail, the risk is background and photographer cues, and evaluation looks at confusion among neighbours. Hierarchical means labels sit in a parent-child taxonomy. Prediction can stop at any depth, the risk is impossible parent-child combinations, and evaluation uses semantic distance. Ordinal means the categories carry a meaningful order. Threshold structure matters, the risk is treating a distant error like an adjacent one, and evaluation must measure ordered deviation.

The ordinal case is the one most often flattened on the way to a decision, and there is a regulatory record of exactly that. In January 2018 the FDA granted a De Novo for IDx-DR. Its summary, DEN180001, documents an ordered clinical severity scale being cut into a binary. Three certified readers graded widefield stereo photographs on the ETDRS scale by majority vote, and the summary states the rule in a single sentence: “Each participant was categorized as mtmDR+ (ETDRS level 35 or higher and /or DME present), or mtmDR- (ETDRS level 10-20 and DME absent).” One cut point, at level 35. Everything the order knew about distance is spent there.

What survives the cut is a binary operating profile. The pivotal trial, published in npj Digital Medicine in 2018, ran 900 participants at 10 primary care sites. Observed sensitivity was 87.4%, observed specificity 89.5%, imageability 96.1%, PPV 72.7% and NPV 95.7%, against pre-specified thresholds of 85.0% sensitivity and 82.5% specificity. Those numbers say whether the system cleared its bar. None of them says how far a wrong grade sat from the right one. That is the question the ordinal structure was there to answer, and it is why the grading scale and the decision rule have to be documented separately.

A multilabel head can estimate each attribute separately. Yet labels may be mutually exclusive, conditionally related, or defined only under certain parent classes.

Structured constraints, conditional heads, or post-processing may make the label set more coherent. They can also encode incorrect assumptions. So check both whether the constraints hold and how well the task itself is done.

One dataset was built in the shape of the real problem. Its authors describe it as “the iNaturalist species classification and detection dataset, consisting of 859,000 images from over 5,000 different species of plants and animals”. The images “have been verified by multiple citizen scientists”. The collection was built to “feature a large class imbalance”, because the natural world is imbalanced, not because anyone failed to balance it. The 2017 release splits into 5,089 categories, with 579,184 training, 95,986 validation and 182,707 test images. The imbalance is a property of a published, countable object, not a claim about data in general.

The headline result names where the difficulty concentrates: “Results show that current non-ensemble based methods achieve only 67% top one classification accuracy, illustrating the difficulty of the dataset. Specifically, we observe poor results for classes with small numbers of training examples suggesting more attention is needed in low-shot learning.” A third of the examples wrong in aggregate, with the failures piled onto the rarest species, is what a target design has to answer for. Structure over the labels is one answer. More supervision for the thin classes is another. They are not the same decision.

Subtle tasks often amplify collection artifacts, because the true signal is difficult. The size of that amplification has been measured. A Caltech team trained species classifiers on camera-trap images, then tested them at camera locations the training set had never seen. Their finding, in 2018: “In our experiments state-of-the-art algorithms show excellent performance when tested at the same location where they were trained. However, we find that generalization to new locations is poor, especially for classification systems.”

The numbers behind that sentence. On full images, top-1 error was 20.83% at trained locations and 41.08% at new ones, a 97% increase. Cropping to bounding boxes cut error at trained locations to 8.14%, but the gap to new locations widened to 140%, with 19.56% error there. Removing the background helped in absolute terms and hurt in relative terms. That is the shortcut showing itself twice.

The same shift turns up under a different metric. WILDS, a Stanford-led benchmark from 2021, scored 182 species across disjoint camera traps: macro F1 fell from 47.0 (±1.3) in distribution to 31.0 (±1.3) out of distribution.

Fine-grained decisions often require both whole-object context and localized detail. Scene context can be useful, and can equally encode collection bias. The object silhouette separates broad categories and pose. Parts and landmarks support the fine distinctions. Texture and markings carry class evidence but are sensitive to resolution. Metadata and history are nonvisual evidence, valuable when available and ethically appropriate.

Choosing a scale is therefore an experiment, not a preference. In the camera-trap study above, cropping from the full image to the animal's bounding box cut error at trained locations from 20.83% to 8.14%. The same pixels. A different decision about which scale the classifier is allowed to see.

Start from label semantics, before choosing output layers.

First, map label relations: mark exclusivity, hierarchy, order, prerequisites, and allowed uncertainty. Second, verify observability. Check which distinctions are visible at the deployed resolution and crop; the 20.83%-to-8.14% swing from cropping is what that check is worth. Third, choose the output structure: compare independent, conditional, hierarchical, and ordinal formulations against the relations you mapped. Fourth, build rare-combination tests, including valid label combinations that training rarely contains. Pneumothorax without a drain is the model case. Fifth, connect outputs to action: set thresholds and fallback behavior for each decision context, and write down where the cut falls, as DEN180001 does at ETDRS level 35.

Micro-averaged metrics emphasize frequent labels. Macro averages give rare labels more influence. Example-based metrics answer a different question: whether a complete label set is useful for one image.

Hierarchical systems should also report coarse accuracy, leaf accuracy, consistency, and semantic distance. A team at Five AI made the last of those concrete in 2020, scoring mistakes as graph distance in a class hierarchy across two real taxonomies: a height-13 WordNet tree over 608 classes (tieredImageNet-H), and the 8-level complete taxonomy over 1,010 classes released for a 2019 fine-grained recognition workshop (iNaturalist-H). Their result is that top-1 error and mistake severity are different quantities, and they have moved differently. ImageNet top-1 error fell sharply across architecture generations. The severity of the remaining mistakes barely moved. “This has led to a situation in which mistakes are less likely to be made than before, but are equally likely to be absurd or catastrophic when they do occur.”

A hierarchy-aware metric is not automatically safer than the metric it replaces. An independent group showed in 2021 that average mistake-severity is exploitable. It divides by the number of misclassified samples, so a hierarchy-aware classifier can lower the average largely by adding extra low-severity mistakes.

The loss is the other half of the same decision. Cui and colleagues attacked the imbalance from the loss side in 2019: “In this work, we argue that as the number of samples increases, the additional benefit of a newly added data point will diminish.” Because samples overlap, they define an effective number of samples that “can be calculated by a simple formula” — (1-β^n)/(1-β). Re-weighting the loss by it is the whole of the method, tested “on artificially induced long-tailed CIFAR datasets and large-scale datasets including ImageNet and iNaturalist”. On iNaturalist 2018, with a ResNet-50 at 224x224, top-1 validation error fell from 42.86% with softmax cross-entropy to 38.88% with class-balanced focal loss (β=0.999, γ=0.5). On iNaturalist 2017 it fell from 45.38% to 41.92%. An independent group later carried that same 42.86% against 38.88% pair into its own table as baselines, which is why the four points are worth quoting at all. Ordinal systems, meanwhile, need metrics that respect how far a prediction moved. The metric you report and the loss you train are two different decisions about which labels matter.