Multilabel, Hierarchical, and Long-Document Classification

The word “classification” hides several incompatible label contracts.

The rest of this lesson works from three real label sets, and no two of them behave alike. EuroVoc, the taxonomy the Publications Office of the European Union maintains for EU legislation, holds roughly 7,000 concepts. Reuters RCV1-v2 carries 103 Topic codes arranged in a hierarchy, and each story is assigned at least one of them and as many as 17. The MIMIC-III discharge summaries used for ICD coding are tagged from 8,922 labels, and a single summary carries a mean of 15.9 of them. None of the three is a choice among mutually exclusive alternatives. Each fails differently when you treat it as one.

Independent binary heads are the simple baseline for a multilabel problem: one classifier per label, each trained as though the others did not exist. The label sets they are trained on are not built that way. And the dependence between labels is not always a fact about the world.

Reuters RCV1-v2 is 804,414 newswire stories labelled from 103 Topic codes arranged in a hierarchy. Lewis and colleagues released it in 2004. scikit-learn, which distributes the collection, records that “Each sample belongs to at least one topic, and to up to 17 topics”. Much of that cardinality is manufactured. Reuters' Hierarchy Policy required that every ancestor of an assigned code be assigned as well, and the authors state the consequence plainly: “Adding ancestor codes creates some very high frequency codes (CCAT is assigned to 46% of the corpus), as well as strong, partially deterministic, dependencies between hierarchically related codes.” A co-occurrence learned from that data may describe the news. It may only describe a coding rule.

The rule was not obeyed. In the original RCV1-v1, 14,786 documents — 1.8% — are missing an ancestor of a code they carry. Every one of the 21 Topic codes that have children is missing somewhere, 25,402 occurrences in all. RCV1-v2 is the repaired version: Lewis and colleagues added those 25,402 ancestor assignments and dropped the 2,377 documents that violated the Minimum Code Policy, leaving 804,414. So a constraint strong enough to define the taxonomy was still broken by 1.8% of the human coding. That is the rate at which a hard-coded version of it would be overwriting a human decision rather than fixing one.

Frequency is the other reason independent heads struggle. EURLEX57K, built by Chalkidis and colleagues in 2019, holds 57k EU legislative documents “with an average length of 727 words”, tagged from EuroVoc. Of EuroVoc's concepts, “only 4,271 (59.31%) are present” in the corpus. Of that subset, “only 2,049 (47.97%) have been assigned to more than 10 documents”. Most of a working taxonomy is tail. An independent head has to learn each tail label alone.

Dependencies can be handled through features, joint models, post-processing constraints, or human review. Each method should preserve a path for legitimate exceptions rather than hard-coding every observed correlation — including the correlations a coding policy, rather than the world, put there.

A hierarchy can support graceful specificity when fine labels lack enough evidence.

EuroVoc is what such a taxonomy looks like when a public institution maintains one. The Publications Office of the European Union calls it “the EU's multilingual and multidisciplinary thesaurus” on its “Browse by EuroVoc” page, and says what is in it: “It contains keywords, organized in 21 domains and 127 sub-domains, which are used to describe the content of documents in EUR-Lex.” Its roughly 7,000 concepts sit beneath those 21 domains and 127 sub-domains. The legislative documents themselves “have been annotated by the Publications Office of EU with multiple concepts from EUROVOC”. The depths below are not a modelling convenience here. They are administrative fact, published and browsable, with a maintainer to answer for them.

The corpus, though, is not the taxonomy. Only 59.31% of EuroVoc's concepts ever appear across the 57,000 documents of EURLEX57K; the rest are dead weight in that collection. A classifier that predicts at every published depth is predicting into branches its training data never populates. The hierarchy will not tell you which branches those are. Only a count over the corpus will.

One pooling rule rarely handles all these patterns equally well, and which pattern a corpus has is measurable before any architecture is chosen.

Two corpora make the point in opposite directions. The MIMIC-III full-label training set used for ICD coding holds 47,724 discharge summaries carrying a mean of 15.9 labels each, drawn from 8,922 labels — 8,921 unique ICD-9 codes, 6,918 diagnosis and 2,003 procedure. The documents are long, and unevenly so. “The number of word tokens per document in the MIMIC datasets is about 1,500 on average and can be greater than 6,500”, Vu and colleagues noted in 2020, and they truncate every document to 4,000 tokens as a consequence. The training-set mean is 1,485 tokens. A 512-token window sees well under half of an average summary and far less of a long one. The label set it must satisfy stays at 15.9 codes per document.

EURLEX57K is the opposite case. Its 57k EU legislative documents come “with an average length of 727 words”, and the evidence in them sits at the front: the header-and-recitals zone “contains enough information and is shorter than 500 tokens in 83% of our dataset’s documents”. That is why Chalkidis and colleagues ran BERT over the first 512 tokens rather than reaching for a long-context architecture. Where the evidence sits was established before the model was chosen.

Increase complexity only when the evidence pattern requires it, and let a measurement rather than a preference decide when it does.

Two 2022 studies settle the question by disagreeing. On EURLEX, a BERT model truncated to the first 512 tokens reaches 73.09 micro-F1 while Longformer reaches 54.53; Park and colleagues measured both. They then built Inverted EURLEX by reversing the order of each document's sections, so that the key information falls at the end. Truncation still wins there, 70.53 against 56.47. On ECtHR the verdict flips. Dai and colleagues find Longformer over 4,096 tokens at 81.0 micro-F1, and a hierarchical model over the same 4,096 at 81.1. The first 512 tokens manage 73.5. That is beaten by 512 tokens chosen at random, at 79.0, and matched by the most informative 512 at only 72.4. Their own conclusion is the one to carry into the ladder: “It is also worthy noting that, among these baselines, there is no single best strategy to choose which 512 tokens to process. Using the first 512 tokens works well on MIMIC-III and Hyperpartisan datasets, but it performs much worse than 512 random tokens on ECtHR.”

Hold the two truncation scores side by side. On EURLEX, taking the first 512 tokens scores 73.09 against Longformer's 54.53. On ECtHR, the same strategy scores 73.5 against Longformer's 81.0. The strategy did not change and its score barely moved. The verdict reversed. Nothing about the architecture predicts which case you are in.

So the diagnostics belong before step 4, not after it. Inverted EURLEX is one you can run on your own corpus: reverse the section order and re-measure, and if the score barely moves, position is not carrying the decision. Random-window sampling is the other. If 512 random tokens beat your first 512, as they do on ECtHR, the front of the document is not where the evidence lives, and a longer context is buying something real rather than buying comfort.

Micro F1 emphasizes frequent label decisions, macro F1 gives labels equal weight, and sample-based measures assess each document's predicted set; hierarchical metrics can give partial credit for a correct parent when a child is missed. On a long-tailed label set these are not shades of emphasis. They are separate verdicts on identical predictions.

The cleanest demonstration is one model scored two ways. CAML, run on MIMIC-III with the full ICD-9 label set, reaches micro-F1 0.539 and macro-F1 0.088. The two averages already differ by almost an order of magnitude on one set of outputs. In 2018 Mullenbach and colleagues, who built the model, priced the cheap strategy: “It is also apparent how difficult it is to achieve high Macro-F1 scores, due to the metric's emphasis on rare-label performance. To put these results in context, a hypothetical system that performs perfectly on the 500 most common labels, and ignores all others, would achieve a Macro-F1 of 0.052 and a Micro-F1 of 0.842.” A system that answers 500 labels and abandons the rest of the 8,921 unique ICD-9 codes beats the real model on micro-F1, 0.842 against 0.539. It loses to the real model on macro-F1, 0.052 against 0.088. Report one average and you can call that system excellent or useless, truthfully, without altering a single prediction.

So also report exact-set accuracy, label cardinality, combination coverage, constraint violations, and abstention. Inspect documents whose decisive evidence appears late or spans several chunks.

Slicing the label set can reverse a verdict about a model in the same way. In EURLEX57K, BERT over the first 512 tokens “outperforms all other methods, even though it considers only the first 512 tokens”. The same run fails at the zero-shot labels, “since it does not have a component that exploits prior knowledge”. One system, two verdicts, decided by which slice of the label set was being scored.

Take a document-tagging problem with at least twelve labels. Draw parent-child relations, valid overlaps, forbidden combinations, and unknown cases.

Then run the three measurements this lesson is made of. First, the RCV1 audit: count the documents in your own gold data that break the hierarchy rule you are about to enforce. Lewis and colleagues found 14,786 documents — 1.8% — breaking Reuters' Hierarchy Policy across 25,402 missing code occurrences. A rate of zero means a script applied the constraint and the data can tell you nothing about it. A non-zero rate names the exceptions a hard constraint would overwrite.

Second, the zone ablation: score the model on each document zone separately and on the whole document, as the EURLEX57K authors did before choosing a 512-token window. Then reverse the section order and re-measure — the Inverted EURLEX test, on which truncation still scored 70.53 against Longformer's 56.47.

Third, the threshold ablation: report micro and macro F1 with tuned per-label boundaries, and again with every boundary left at 0.5. Edin and colleagues saw CAML fall from 55.4/20.4 to 53.3/17.1 on that one change. Finally, review five cases where a constraint would reject a legitimate exception, and check whether those cases sit among the rare labels the 0.5 default has already silenced.