Sequence Labeling: POS, Named Entities, and Span Boundaries

Several mappings must remain consistent before a highlighted entity reaches a user: source characters, normalized and segmented text, model tokens or subwords, sequence labels, and reconstructed spans.

Every one of those joints has been measured by somebody, and the measurements are the substance of this lesson. The tag scheme is worth 1.42 F1 on one test set, and loses on the development set of the same corpus. The reference labels against which the whole pipeline is scored carry mistakes in about 5.38% of CoNLL-2003 English test sentences. The unit that step one calls a character resolves to seven of one thing and four of another for a single Hindi word, depending on which standard does the counting. Where the sections below could assert that alignment matters, they give the figure instead.

The scheme is a serialization choice rather than the entity definition itself. It is also a choice with a measured price. The two papers that measured it do not agree on the sign.

One of them found the richer scheme wins, and put it in the abstract: “We find that BILOU representation of text chunks significantly outperforms the widely adopted BIO.” Ratinov and Roth's table gives the size of it. Their end system scored 89.15 F1 with BIO and 90.57 with BILOU on the CoNLL-2003 English test set, a gain of 1.42. On MUC-7 dev it was 86.76 against 88.09, on MUC-7 test 85.15 against 85.62. Then the order reverses. On the CoNLL-2003 development set BIO scored 93.61 against BILOU's 93.28 — the other split of the very corpus that supplies the headline gain. The same paper counts what the richer scheme costs the decoder. With the BILOU encoding of four entity types each token can take 21 states. Greedy decoding needs 21 comparisons per token; Viterbi needs 21³.

Ten years earlier the comparison had come out the other way, on chunking. Tjong Kim Sang and Veenstra tested seven representations for baseNP chunking in 1999 — IOB1, IOB2, IOE1, IOE2, IO, [+] and [ + IO, IO +]. The best score belongs to IOB1, at Fβ=1 92.37 against 91.78 for IOB2. That is a spread of 0.59 points across seven formats. The authors concluded that the differences between the formats were not significant.

So the honest summary is neither “BILOU is better” nor “the scheme does not matter”. The choice is worth about a point when it is worth anything. It can go the wrong way on a second split of the same data. And it multiplies the state space the decoder has to search. Pick the scheme for what your spans must support — adjacent mentions, single-token mentions, overlap — then measure it on your own corpus rather than inheriting either verdict.

“Paris” can be a location, an organization name, a person surname, or part of a product; whether “University of Bologna” includes “University” and whether dates include prepositions depend on annotation rules.

Entity types should correspond to a downstream use such as search, linking, redaction, or case creation; guidelines need boundary rules, nesting policy, ambiguity handling, and examples from the actual domain.

Two decades of named-entity numbers were measured against a schema that two people wrote down for the CoNLL-2003 shared task. It has four types: “persons, locations, organizations and names of miscellaneous entities that do not belong to the previous three groups”. Tjong Kim Sang and De Meulder took the English data from the Reuters Corpus and the German from the ECI Multilingual Text Corpus, the German portion “extracted from the German newspaper Frankfurter Rundshau”. The English training set is 946 articles, 14,987 sentences and 203,621 tokens. A further 3,466 sentences were held out for development and 3,684 for test. Four types over two newswires — a product ontology, chosen by two people in 2003.

The same kind of choice, made about parts of speech, has been made at both extremes for the same language. The Penn Treebank — “over 4.5 million words of American English” — was built on a deliberately pared-down inventory, and Marcus and colleagues said so in 1993: “The Penn Treebank tagset is given in Table 2. It contains 36 POS tags and 12 other tags (for punctuation and currency symbols).” Pared down from what is the interesting part. The Brown Corpus had 87 simple tags plus compounds, the LOB Corpus roughly 135, the Lancaster UCREL group roughly 165, and the London-Lund Corpus of Spoken English 197. Nineteen years later Petrov and colleagues went the other way. They proposed “a tagset that consists of twelve universal part-of-speech categories” plus “a mapping from 25 different treebank tagsets”, yielding data for 22 languages. Twelve categories or 197: English did not change between those two documents.

At the far end, an entity schema can be a legal instrument. The HIPAA Privacy Rule's Safe Harbor method, 45 CFR §164.514(b)(2)(i), enumerates exactly 18 lettered categories of identifier, (A) through (R). All of them must go before health information stops being individually identifiable. The rule introduces them in the flat voice of regulation: “The following identifiers of the individual or of relatives, employers, or household members of the individual, are removed:”. The boundary rules are written into the schema rather than left to an annotator. (B) permits keeping the first three digits of a ZIP code only where the combined area “contains more than 20,000 people”, and otherwise requires those digits to be changed to 000. (C) removes all elements of dates except the year and folds every age over 89 into “age 90 or older”. (R) is an open-ended catch-all, “Any other unique identifying number, characteristic, or code”. A different federal agency describes the same provision the same way. NIST Special Publication 800-188, from September 2023, says Safe Harbor works “by specifying that health information is considered to be de-identified through the removal of 18 kinds of identifiers”, and adds that it still “retains some risk of identification”. Four types chosen by two researchers in 2003, 36 tags chosen in 1993, twelve categories proposed for 22 languages, 18 categories written into the Code of Federal Regulations: the annotation guideline is the same kind of document every time. Only in the last case is it also law.

Inspecting them separately often points to different fixes, because the families have different owners: the annotation guideline, the tokenizer, the decoder, the post-processor.

They also carry different prices, and the price is set by the consumer rather than by the metric. Under Safe Harbor's case (C), all elements of a date except the year must be removed. So a redaction span that covers the month but misses the leading day is a boundary-only error with almost complete character overlap — and the record still carries an element the rule requires to be gone. A type-only error on the same span is invisible to the redaction and fatal to the search index. Count the six families separately. Then price each one against what the downstream system actually does with the span.

A decoder or post-processor should enforce only rules justified by the schema. Each of the five steps has a documented failure behind it rather than a general worry.

Legal transitions are a property of the scheme's state space. Ratinov and Roth count 21 states per token for the BILOU encoding of four entity types, so greedy decoding needs 21 comparisons per token where Viterbi needs 21³. Subword merging applies the policy you declared — the first-sub-token rule, or whichever one you wrote down — rather than the library default. Mapping to source offsets means committing to a unit, and Unicode Standard Annex #29 is explicit that a “user-perceived character” is often several code points.

Applying schema constraints means applying the constraints someone actually wrote. Under 45 CFR §164.514(b)(2)(i), the first three digits of a ZIP code may be kept only where the combined area “contains more than 20,000 people”, and must otherwise be changed to 000. Every age over 89 becomes “age 90 or older”. Those are post-processing rules, not model behavior. They belong in tested code rather than in a guideline nobody executes.

The last step is the one most often dropped. Even a correctly applied schema leaves residual risk: NIST Special Publication 800-188 notes that Safe Harbor “retains some risk of identification”. That is the argument for routing uncertain high-consequence spans to review instead of forcing a value out of the decoder.

Exact span F1 requires both boundary and type to match. Partial-overlap scores can reveal near misses. But generous matching may overstate utility when downstream systems need exact redaction or database keys.

Report boundary, type, length, nesting, language, and document-source slices, and for part-of-speech tagging, examine confusion among categories and unknown or code-switched forms.

The CoNLL-2003 organisers refused to report a rate without an error bar. Sixteen systems took part and every one beat the baseline, which “only identified entities which had a unique class in the training data”. For a phrase inside more than one entity, the baseline “would select the longest one”. Significance came from bootstrap resampling: “From each output file of a system, 250 random samples of sentences have been chosen”. Performance A counted as different from B only when A fell outside the central 90 per cent of B’s distribution. On that test the best English system was not significantly ahead of the next one.

Under that ceiling there is a second one: the reference. Six researchers audited the corpus in 2019 and put the result in their abstract: “We are able to identify label mistakes in about 5.38% test sentences, which is a significant ratio considering that the state-of-the-art test F1 score is already around 93%.” They released a manually corrected test set. A team at IBM arrived at the same place by another route, using semi-supervised methods on the same corpus, and reported “over 1300 incorrect labels (out of 35089 in the corpus)”. The number of incorrect labels in the test fold is comparable to the number of errors state-of-the-art models make on it. Two teams, two methods, one conclusion: part of what the scoreboard charges to the model belongs to the annotation.

Part-of-speech tagging has the same ceiling, and the Penn Treebank team measured it on themselves. Four linguistically trained annotators worked over eight 2,000-word Brown Corpus samples in two modes: “Mean inter-annotator disagreement was 7.2% for the tagging task and 4.1% for the correcting task (with medians 7.2% and 3.6%, respectively)”. Correcting an automatic tagger's output ran at 20 minutes per 1,000 words, against 44 for tagging from scratch. Eighteen years later Manning classified a sample of 100 tagger errors from Penn Treebank section 19. 28.0% were “Inconsistent/no standard”, a further 15.5% “Gold standard wrong” — two classes that together “comprise over 40% of the data”. The state of the art was about 97.3% token accuracy and 56% sentence accuracy. His conclusion: “The easiest path for continuing to improve POS tagging seems to be to look at the cases in classes 6 and 7, where the gold standard data is just wrong or is inconsistent because of the lack of clear tagging guidelines.”

Partial credit is not free to grant either. A span that overlaps a required identifier across most of its characters has not removed it. A partial-overlap metric awards that prediction a fraction of a point; Safe Harbor awards it nothing. Choose the metric's tolerance to match the consumer, then report the exact-match number beside it.

Write fifteen examples containing adjacent entities, punctuation, abbreviations, nested names, Unicode combining marks, and subword-heavy identifiers. Annotate source character offsets and expected types — and state in the fixture which unit “character” means, code point or grapheme.

Make four of the fifteen do specific work. Put यूनिकोड in one and assert both counts, seven Unicode code points and four graphemes, so the fixture fails loudly the day the runtime changes unit. Put a mention nested two deep in another: the Wall Street Journal re-annotation found up to 6 layers of nesting in newswire, and a flat reconstruction should raise rather than silently flatten. Put a date, a ZIP code and an age in a third, and run the Safe Harbor rules as post-processing — year only, three ZIP digits only where the area “contains more than 20,000 people”, ages over 89 as “age 90 or older”. Encode a fourth example in both BIO and BILOU and assert the reconstructed spans are identical. A serialization that changes the spans is a bug. The 1.42-point gap Ratinov and Roth measured is a modeling result, not a licence for the decoder to disagree with itself.

Run the complete preprocessing and reconstruction path. Then check both rendered highlights and machine-readable offsets. Record whether each error began in annotation, tokenization, model output, or post-processing. Keep a fifth bucket for the reference itself, because on CoNLL-2003 about 5.38% of the test sentences carried label mistakes of their own.